11institutetext: The University of Sydney, Australia 22institutetext: JD
Explore Academy, China 33institutetext: Nanyang Technological University,
Singapore
33email: {sihan.ma<EMAIL_ADDRESS>33email:
<EMAIL_ADDRESS>33email<EMAIL_ADDRESS>
https://xymsh.github.io/RICH-CAT

# Contact-aware Human Motion Generation from Textual Descriptions

Sihan Ma 11 Qiong Cao 22 Jing Zhang 11 Dacheng Tao 33

###### Abstract

This paper addresses the problem of generating 3D interactive human motion
from text. Given a textual description depicting the actions of different body
parts in contact with objects, we synthesize sequences of 3D body poses that
are visually natural and physically plausible. Yet, this task poses a
significant challenge due to the inadequate consideration of interactions by
physical contacts in both motion and textual descriptions, leading to
unnatural and implausible sequences. To tackle this challenge, we create a
novel dataset named RICH-CAT, representing “Contact-Aware Texts” constructed
from the RICH dataset [16]. RICH-CAT comprises high-quality motion, accurate
human-object contact labels, and detailed textual descriptions, encompassing
over 8,500 motion-text pairs across 26 indoor/outdoor actions. Leveraging
RICH-CAT, we propose a novel approach named CATMO for text-driven interactive
human motion synthesis that explicitly integrates human body contacts as
evidence. We employ two VQ-VAE models to encode motion and body contact
sequences into distinct yet complementary latent spaces and an intertwined GPT
for generating human motions and contacts in a mutually conditioned manner.
Additionally, we introduce a pre-trained text encoder to learn textual
embeddings that better discriminate among various contact types, allowing for
more precise control over synthesized motions and contacts. Our experiments
demonstrate the superior performance of our approach compared to existing
text-to-motion methods, producing stable, contact-aware motion sequences. Code
and data will be available for research purposes.

###### Keywords:

3D Human motion Text-to-motion generation Contact modeling.

Figure 1: We address the problem of text-driven 3D interactive human motion
generation from both data and algorithmic perspectives. RICH-CAT is a novel
dataset featuring (a) high-quality motion, (b) accurate contact labels, and
(c) interactive textual descriptions that specify different body parts
interacting with various objects. Using it, we introduce a novel approach
named CATMO to learn the complex human motion dynamics that incorporate
interaction semantics provided by contact, producing natural and plausible 3D
human motions of action (d).

##  1 Introduction

Synthesizing 3D human motion that makes contact with its surrounding
environment is pivotal for simulating daily behaviors that individuals perform
in the real world such as exercising or cooking. This is particularly critical
in a variety of applications including humanoid robots, AR/VR and action
games. The creation of interactive human behaviors from textual descriptions
not only offers significant convenience but also extends the boundaries of
these applications.

This paper addresses the problem of text-driven 3D interactive human motion
generation. Given a textual description illustrating the actions of various
body parts in contact with objects, we synthesize sequences of visually
natural and physically plausible 3D body poses. The problem of text-driven
motion generation [12, 1, 41, 50, 49, 42, 3] has seen considerable research
interest. However, existing methods exploring this domain lack comprehensive
modeling of human motion with contacts in the real world, e.g., synthesizing
only hand-grasping actions that make contact with small dynamic objects [10],
limiting to a few types of predefined contact such as sitting on a chair [49,
42, 3] or overlooking the vertex-level contact labels by only considering
collision [45]. In contrast, our focus is on a significantly more complex
task, which requires accurate capturing of human movement in alignment with a
textual description detailing the action types, interacting 3D body parts and
objects.

To achieve this, we must learn a cross-modal mapping between text and
interactive body motion, which is very challenging in practice for several
reasons. First, there is a dearth of datasets that provide rich contextual 3D
human motion interacting with various objects, accompanied by textual
descriptions and accurate vertex-level contact labels. Second, considering the
intricacy of body parts physically interacting with objects, depicting 3D
interactive human actions based on text descriptions necessitates more than
simple posture descriptions such as “leans on the fence”. A more detailed
illustration like “Leaning on the fence with the left hand” conveys the
details. Lastly, as different body parts correspond to varying degrees with
textual descriptions, it is difficult to model the cross-modal mapping and
generate natural interactive body motions without explicit contact modeling.

We address the above challenges from both data and algorithmic perspectives
and learn to model the complex interaction dynamics in a data-driven way.
First, to overcome the issue of data scarcity and limited posture
descriptions, we expand upon the RICH dataset [16] by building a new set
called RICH-CAT, featuring over 8,500 complex human-object motion and contact
sequences belonging to 26 different in/outdoor action types. Each sequence is
paired with a detailed description, generated by our automatic textual
description generation technique, detailing different body parts engaged in
interacting with objects. This provides a more precise portrayal of
interactive human actions.

Then, we investigate the translation from text to 3D interactive human motion.
We propose CATMO, a novel approach that generates realistic 3D human motion in
close contact with objects from a detailed textual description. Specifically,
we utilize two independent VQ-VAEs to encode the motion and contact modalities
into distinct yet complementary latent spaces, thereby capturing more
intricate representations of each modality and providing each aspect with full
expression. Next, to predict the 3D interactive human motion, we propose an
intertwined GPT architecture that produces motion and contact in a twisted
auto-regressive way, explicitly incorporating contact information into the
motion generation process. With the learned Motion VQ-VAE decoder, the output
from intertwined GPT is decoded, yielding a sequence of 3D poses with
physically plausible contacts. Furthermore, we present a pre-trained text-
motion model that effectively decomposes interactive textual descriptions and
aligns them with the generated motion sequence by alignment loss.

To the best of our knowledge, our proposed method is the first approach to
explicitly address the problem of generating realistic 3D interactive human
motion from textual descriptions depicting detailed interactions by contacts.
We quantitatively evaluate the realism and diversity of our synthesized motion
compared to ground truth and existing methods. Both quantitative and
qualitative results demonstrate the superior performance of our approach,
which is capable of generating stable, contact-aware motion sequences from
texts. Additionally, we show the adaptability of our model for applications in
human-object interaction synthesis in static scenes.

##  2 Related Work

Table 1: Comparison of various text-to-motion datasets. “Text: Body part” and
“Text: Object” refer to the provision of information in textual descriptions
about human body parts and the objects involved in contact within each
sequence. “Text: Per-frame” denotes the precise alignment of text annotations
with the motion intervals in the sequence. “Contact” indicates the
availability of accurate contact labels. “Real scene” distinguishes whether
the motion was captured in real-world settings or within controlled lab
environments. “GT motion” means whether the human motion is mocap or pseudo-
GT. “#Actions” indicates the number of action categories.

Dataset | Text: Body part | Text: Object | Text: Per-frame | Contact | Real scene | GT motion | #Actions
---|---|---|---|---|---|---|---
HumanML3D [12] | ✗ | ✗ | ✗ | ✗ | ✗ | Mixed | -
KIT-ML [29] | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | -
BABEL [30] | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | 260
HUMANISE [45] | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | 4
HumanAct12 [13] | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 12
UESTC [17] | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 40
NTU-RGB+D [21] | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 120
RICH-CAT | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 26

####  2.0.1 Text-driven human motion generation.

Text-driven motion generation synthesizes 3D human motion from textual
descriptions. There are two main categories of methods: latent-space [45, 1,
9, 27, 12, 41, 18, 22] and diffusion-based [50, 42, 3, 4, 19] approaches. In
the former category, Language2Pose [1] introduced shared embeddings for
language and pose, while [9] extended this concept with a two-manifold space
design. TEMOS [27] used a transformer structure to encode motion into a single
embedding to avoid a tedious auto-regressive process. To address fixed-length
prediction issues, [12] introduced a text2length module for variable-length
motion generation. A recent straightforward text-to-motion framework [49]
overcomes these challenges by mapping motion to a discrete latent space and
aligning motion and text embeddings with a GPT-like model. Corrupted input
training strategies and casual self-attention are adopted to ensure variable
motion length.

Regarding diffusion models [40, 36, 37, 32], initially developed for
generating complex images, they have demonstrated their effectiveness in
various domains, including conditional motion synthesis. MotionDiffuse [50]
and MDM [42] were among the first to employ diffusion models in this context.
However, diffusion models are associated with significant computational
overhead, intricate sampling procedures, and the need for carefully designed
motion noises. An attempt to address motion noise and temporal inconsistency
challenges was made by [3] by projecting raw motion data into a latent space.
While effective, the majority of diffusion-based methods exhibited limited
improvements in performance and efficiency compared to traditional approaches
[49]. While prior methods aim to explore the structure design, they neglect
the impact of contact when generating motion from text. Our research pioneers
the consideration of human-object contact in text-driven motion generation,
introducing a novel contact-aware text-to-motion dataset and approach to fill
this gap.

####  2.0.2 Contact-aware motion models.

Human-object interaction, specifically in terms of human-object contact, plays
an essential role in understanding and generating natural human actions and
behaviors. This aspect has been considered in various domains such as motion
reconstruction, motion priors, and motion understanding. In motion
reconstruction, many methods [46, 35, 39] address feet-ground contact by
training contact detectors, while others leverage physical simulators [23, 48,
8, 11, 15] and optimization goals inspired by physics principles [5, 38].
Motion prior models [51, 34, 24] often incorporate human-ground contact beyond
the ordinary motion representation to model the transition of human movements.
This offers optimization objectives that enhance motion generation and
reconstruction toward a physically plausible direction. In motion
understanding, some methods [43, 16] attempt to infer 3D vertex-level contact
on a human body from 2D images, aiming to understand how humans interact with
their surroundings. Despite these efforts, the significance of human-object
contact has remained relatively unexplored in the context of text-driven human
motion generation. To fill this gap, we introduce a novel contact-aware
dataset and a simple and effective method to advance research in this domain.

####  2.0.3 Text-to-motion datasets.

Before commencing our work, we require a dataset that encompasses 3D human
motions, precise contact labels, and corresponding textual descriptions.
Existing text-to-motion datasets have been summarized in Table 1. Datasets
like HumanAct12 [13], UESTC [17], and NTU-RGB+D [21] cover a wide range of
daily actions but offer only basic action labels. HumanML3D [12], KIT-ML [29],
and BABEL [30] are well-known text-to-motion datasets, featuring extensive
mocap motions. BABEL [30], in particular, includes frame-level textual
descriptions. However, these datasets capture data in controlled lab settings
and primarily focus on poses and self-contact, often neglecting interactions
with other objects. The HUMANISE [45] dataset, while providing 3D scene
meshes, is limited to just four indoor actions, and its motion data lacks
fine-grained contact information between humans and objects. To address these
limitations, we have developed a novel text-to-motion (T2M) dataset based on
the RICH [16] dataset, including various motion dynamics captured from daily
actions. This new dataset contains motion data captured in real-world scenes,
diverse indoor and outdoor actions, and accurate vertex- and joint-level
contact labels. Furthermore, we have enriched it with frame-level textual
annotations that specify interaction between body parts and objects.

####  2.0.4 Human-object interaction generation.

Human-object interaction (HOI) generation [2, 47, 20, 7, 26] aims to produce a
holistic human and object motion sequence conditioned on the object’s
geometry, sparse object motion evidence, and optionally a textual description.
Different from HOI, our task focuses on modeling the alignment between textual
descriptions and human motion, particularly the varied interactions between
human body parts and objects as detailed in the text. Furthermore, our method
exhibits versatility in adapting to HOI generation in static scenes. Through
the integration of a conditioning module for object geometry, our approach
adeptly generates natural and physically plausible motion sequences that
interact with the given static objects.

##  3 RICH-CAT Dataset

To advance research of text-driven 3D interactive human motion generation, we
create a comprehensive dataset RICH-CAT. This dataset is constructed upon the
RICH dataset [16], which provides high-quality motion capture data for various
human movements. We automatically generate detailed, interaction-aware textual
descriptions. The resulting dataset comprises 8,566 motion sequences. Each
sequence comes with precise vertex-level contact labels and interaction-aware
natural language descriptions that detail the interactions between various
body parts and different objects. Direct comparisons to the existing text-to-
motion datasets can be found in Table 1.

Our automated annotation pipeline (Section 3.1) produces intricate interaction
descriptions and eliminates the need for time-consuming manual labeling. We
also carefully sample and process the motion and contact data for varying
lengths and multi-action scenarios (Section 3.2). Further details on motion
and contact representation can be found in Section 3.3.

###  3.1 Textual Description Generation

In this process, when presented with a sequence of paired (motion, contact)
data and its corresponding 3D scene mesh, our annotation pipeline produces a
templated description containing information about actions and interaction
details regarding human body parts and objects.

####  3.1.1 Annotations for individual frames.

First, we deal with the individual frames. Given a single frame of motion and
contact data, along with the 3D scene mesh, our process begins by labeling all
the objects that could potentially interact with the human body. Then we
assign contact body part labels using the dense vertex-level contact labels
available in the RICH dataset [16]. Additionally, interaction object labels
are determined based on the sorted distance between contacted vertices and
each object in the scene.

####  3.1.2 Aggregating across multiple frames.

With annotations of multiple individual frames, we aggregate them to generate
the textual descriptions for a complete sequence by action grouping,
filtering, and ordering. Initially, we create a unique set of actions for each
sequence by eliminating redundancy. These action types are categorized into
two groups: torso actions and upper-body actions. In some instances, a clip
may contain an excessive number of actions, such as more than two torso
actions or more than three upper body actions. In such cases, we opt to
exclude these motion clips to maintain a more concise and focused dataset.
Additionally, actions of very brief duration, less than five frames each, are
retained in the clips as transitions. But they do not receive textual
annotations. After the filtering process, we arrange the actions in order
based on their start and end indices. If multiple actions occur simultaneously
with significant overlap, we assign them a random ordering.

####  3.1.3 Final automatic textual description generation.

For every clip, when provided with a set of actions, labels for interaction
body parts, and interaction objects, we design a template to automatically
generate textual descriptions following [45]. The template follows this
structure: “$<$Person$>$ $<$action$>$ $<$contact relation$>$ $<$interaction
objects$>$ $<$interaction method$>$ $<$interaction body parts$>$”. The order
may vary slightly to align with typical expressions in natural language. In
cases of multiple actions, this pattern is repeated and linked together using
terms such as “and” or “while”.

###  3.2 Data Sampling

Before the textual annotation process, we must decide what kind of motion data
will be included in our RICH-CAT dataset. To achieve this, we organize all
motion sequences based on their action types and scenes. Subsequently,
sequences lacking meaningful actions are excluded. The remaining data is then
divided into the train, test, and validation sets, approximately at a ratio of
6:1:1. To accommodate variable length conditions in the text-to-motion task,
we segment all the data into clips of various lengths, specifically 64, 128,
and 192 frames. The quantity of them maintains an approximate ratio of
1:0.8:0.6 in each split. The aforementioned textual annotation process is
subsequently applied to each clip to generate the final descriptions.
Furthermore, since the quantity of different action types may vary, we adjust
the sampling probability and overlap size to ensure a balanced ratio.

###  3.3 Motion and Contact Representation

The original human motion data is represented using a parametric body model
known as SMPL-X [25]. This model allows for the calculation of the positions
of 10,475 vertices and 54 joints, using parameters for pose ($\theta$), shape
($\beta$), and facial expression ($\phi$). From these, we select the positions
of the first 22 joints and compute 263-dimensional motion features following
[12].

As for contact labels, binary labels are provided to indicate whether a vertex
is in contact or not, for all 10,475 vertices. To maintain consistency with
the motion features, we convert them into 22 joint-level contact labels using
the regressor provided by the SMPL-X body model [25].

##  4 Method

Figure 2: Architecture of our approach CATMO for text-driven 3D interactive
human motion synthesis. Our model consists of independent (a) Motion VQ-VAE
and (b) Contact VQ-VAE to encode the motion and contact modalities into
distinct latent spaces. Subsequently, we autoregressively predict a
distribution of motion and contact from the text via (c) the intertwined GPT
to explicitly incorporate contact into motion generation. The output from the
intertwined GPT is then fed into the learned Motion VQ-VAE decoder
$\mathcal{D}$ to yield a sequence of 3D poses with physically plausible
interactions. Additionally, the text embedding is extracted from our
pretrained text encoder $\mathcal{E}_{t}$, with an alignment loss ensuring the
consistency between interactive text embeddings and the generated poses.
$\mathcal{E}_{m}$ is the movement encoder pretrained with the text encoder
$\mathcal{E}_{t}$ to calculate the alignment loss.

We aim to generate contact-aware motion sequences conditioned on a textual
description that encompasses the interaction details such as interaction body
parts and objects. Figure 2 illustrates the overall method. Concretely, two
independent VQ-VAEs are used to encode the motion and contact modalities into
distinct yet complementary latent spaces, capturing intricate representations
of each modality (Section 4.2). Then, we propose an intertwined GPT
architecture that generates motion and contact in a twisted auto-regressive
way from the text (Section 4.2) to explicitly incorporate contact into motion
generation. Subsequently, the output from intertwined GPT is fed into the
learned Motion VQ-VAE decoder to yield a sequence of 3D poses with physically
plausible interactions. Lastly, a pre-trained text-motion model is introduced
to effectively decompose interactive textual descriptions and align them with
the generated motion sequence by alignment loss (Section 4.3).

###  4.1 Motion and Contact VQ-VAEs

Considering continuous motion features and binary contact labels as distinct
modalities, we adopt two separate VQ-VAE models [44] to encode motion and
contact modalities into distinct yet complementary latent spaces. The key
insight for the separated modeling is to capture the inherently diverse and
sophisticated patterns of each modality and give each aspect full
expressiveness to facilitate the subsequent generative process.

Specifically, each VQ-VAE model consists of an encoder $\mathcal{E}$, a
decoder $\mathcal{D}$, and a codebook $\mathcal{B}$. The encoder $\mathcal{E}$
first applies a 1D convolution (Conv1D) layer to the motion sequence
$M=[m_{1},m_{2},\cdots,m_{T}]$ or contact sequence
$C=[c_{1},c_{2},\cdots,c_{T}]$ along the temporal dimension to compress by a
ratio of $l$. A series of Conv1D layers and residual blocks follow to convert
the sequence of data into latent feature $Z=[z_{1},z_{2},\cdots,z_{L}]$ of
length $L=T//l$ and dimension $D$. Then the quantitized latent feature
$\hat{Z}=[\hat{z}_{1},\hat{z}_{2},\cdots,\hat{z}_{L}]$ is collected by
fetching the most similar discrete elements in a learned codebook
$\mathcal{B}=\\{b_{k}\\}_{k=1}^{K}$ by
$\hat{z}_{i}=\arg\min_{k}||z_{i}-b_{k}||^{2}$. Finally, the reconstructed
motion $\hat{M}=[\hat{m}_{1},\hat{m}_{2},\cdots,\hat{m}_{T}]$ or contact
$\hat{C}=[\hat{c}_{1},\hat{c}_{2},\cdots,\hat{c}_{T}]$ sequence is recovered
by decoder $\mathcal{D}$.

For each VQ-VAE, we train the encoder, decoder, and codebook simultaneously
with the following three components: reconstruction loss $\mathcal{L}_{re}$,
embedding loss $\mathcal{L}_{e}$, and commitment loss $\mathcal{L}_{c}$. Both
Motion VQ-VAE and Contact VQ-VAE are trained independently and share the same
loss functions for embedding loss and commitment loss, as expressed in the
following:

$\mathcal{L}_{e}=||SG(Z)-\hat{Z}||_{2},\ \
\mathcal{L}_{c}=||Z-SG(\hat{Z})||_{2},$ (1)

where $SG$ is stop-gradient. $Z$ and $\hat{Z}$ are the ground truth latent
feature and the predicted one, respectively.

For the reconstruction loss $\mathcal{L}_{re}$, we employ Binary Cross-Entropy
(BCE) loss to the output logits $\hat{C}$ after softmax operation:

$\mathcal{L}_{re}=-\frac{1}{T*J}\sum_{t,j}((1-C_{t,j})\log(1-\hat{C}_{t,j})+C_{t,j}\log(\hat{C}_{t,j})),$
(2)

where $T$ is the total length of the sequence, and $J$ is the number of
joints. For Motion VQ-VAE, we apply $l1$-smooth loss to both motion feature
and motion velocity following [49], which is omitted here.

The loss for each VQ-VAE is summarized as follows. Note that $\alpha$ and
$\beta$ are the hyper-parameters for balancing loss items.

$\mathcal{L}_{vq}=\mathcal{L}_{re}+\alpha\mathcal{L}_{e}+\beta\mathcal{L}_{c}.$
(3)

For more training details, please refer to supplementary materials.

###  4.2 Intertwined GPT

####  4.2.1 Intertwined motion and contact GPT.

After we learn the separate quantized codebooks, any motion or contact
sequence ($C$ or $M$ respectively) of length $T$ can be represented as a
quantitized latent feature $\hat{Z}$ of $L=T//l$ units and yields the index
sequence $S$ of the same length through quantization. With the quantized
motion and contact representation, we now consider generating interactive
motion from text. Typically, it is formulated as a next-index prediction
problem [49]. Given indices for $i-1$ units $S=[s_{1},s_{2},\cdots,s_{i-1}]$
and the input text embedding (Section 4.3), the goal is to predict the next
index $s_{i}$, and the learned Motion VQ-VAE decoder $\mathcal{D}$ is then
used to reconstruct the motion sequence $\hat{M}$ from the generated indices.

To generate interaction-aware motion sequences, we introduce an interleaved
approach that seamlessly integrates contact token indices
$S^{c}=[s^{c}_{1},s^{c}_{2},\cdots,S^{c}_{L}]$ derived from the same video
clip into the generation process. Instead of treating contact tokens as
additional input features, we capitalize on the temporal correlations between
motion and contact sequences by inputting intertwined motion and contact
sequences $X=[s_{1},s^{c}_{1},s_{2},s^{c}_{2},\cdots,s_{i-1},s^{c}_{i-1}]$,
where the motion and contact indices for each frame are positioned side by
side along the temporal dimension. This allows the model to learn to separate
the two types of token indices and predict the next motion index $s_{i}$ or
contact index $s_{i}^{c}$ conditioned on both past motion and contact indices.
The joint probability of the whole sequence is modeled as follows:

$\begin{split}p(S_{1:L},S^{c}_{1:L}|Z_{t})=\prod_{i=1}^{L}&p(s_{i}|s_{<i},s^{c}_{<i},Z_{t})p(s^{c}_{i}|s_{\leq
i},s^{c}_{<i},Z_{t}),\\\ \end{split}$ (4)

where $Z_{t}$ is the text embedding extracted by our interaction-aware text
encoder introduced in Section 4.3. During inference, the motion and contact
sequences will be generated in an autoregressive manner conditioned on text
embedding $Z_{t}$. There will be two cases to consider: 1) predicting the
next-unit motion index $s_{i}$ based on motion and contact indices of past
units $s_{<i},s^{c}_{<i}$; 2) inferring the current-unit contact index
$s^{c}_{i}$ based on the current-unit motion index $s_{i}$ and all previous
indices $s_{<i},s^{c}_{<i}$. For each step, there are no specific mode flags
to indicate the case. This design, combined with the multi-modal corruption
training strategy introduced later, effectively leverages the complementary
nature of motion and contact within the same frame and the continuity across
different frames, leading to more stable and temporally consistent outcomes.

####  4.2.2 Multi-modal corruption training strategy.

Generally, to reduce the discrepancy between training and testing, a common
practice is to randomly corrupt a portion of the input sequence during
training to enhance model robustness. In the case of simultaneously inputting
motion and contact indices, a reasonable approach would be to randomly replace
motion and contact indices $[s_{i},s^{c}_{i}]$ for certain frames with random
values. However, this practice might cause the model to neglect the scenario
where the current-unit contact index $s_{i}^{c}$ is generated based on the
history of indices and the current-unit motion index
$S=[s_{1},s^{c}_{1},\cdots,s_{i}]$. Consequently, the connection and
interaction between motion and contact may diminish. Therefore, we adopt a
mixed-random strategy, treating each token index equally and randomly
corrupting a portion during training.

####  4.2.3 Objective functions.

Our objectives encompass two aspects: the reconstruction capability for
corrupted indices and the next motion/contact index, as well as the alignment
between text and motion. For the former, we employ Cross-Entropy (CE) loss.
Regarding the latter, we calculated the contrastive loss between the
embeddings of text and the recovered motion sequence. First, the motion
sequence is recovered from the predicted indices via the Motion VQ-VAE
decoder. Then the motion embedding is extracted using the pre-trained motion
model mentioned in Section 4.3. Finally, we calculate contrastive loss with
the input text embedding.

###  4.3 Interaction-aware Text Encoder

For text embedding, recent methods [41, 18, 49] have utilized language model
text encoders like CLIP [31] and sBERT [33]. While these models exhibit good
generalization, they are not specifically finetuned on datasets containing
descriptions of interactive motions. We observed that CLIP struggles to fully
comprehend textual information with interactions, assigning high similarity to
descriptions of different interactions but similar actions. For instance,
descriptions like “push up on the ground” and “push up on the fence” might
differ by only one word, leading to minimal distinction in text embeddings for
CLIP. However, in motion, these actions exhibit significant differences in
global orientation and position. This discrepancy significantly affects
retrieval results.

Inspired by [28], we employ a straightforward pretrained text-motion matching
model designed to align interaction components between text and motion. Our
text-motion matching model architecture is straightforward, comprising a text
encoder to extract textual embeddings, and a movement encoder to subsequently
extract interactive motion embeddings. The embeddings from both modalities are
aligned through contrastive loss. After training, the frozen text encoder is
used to extract text embeddings as inputs for the generative model. Please
refer to supplementary materials for more details.

##  5 Experiment

###  5.1 Experimental Settings

####  5.1.1 Comparison methods.

We evaluate our method CATMO and three state-of-the-art text-to-motion methods
on our RICH-CAT dataset, including one latent-space method T2M-GPT [49] and
two diffusion-based methods MDM [42], MLD [3]. For a fair comparison, we
modified the above methods by directly concatenating the contact features with
the motion inputs, denoting MDM*, T2M-GPT*, MLD*. All the methods are trained
on our RICH-CAT training set and evaluated on the test set. More details are
provided in supplementary materials.

####  5.1.2 Evaluation metrics.

Following [12], we pretrained an evaluation model on the RICH-CAT training
set. Motion and text embeddings are derived from it and assessed using five
commonly used metrics. Frechet Inception Distance (FID) measures the distance
between the generated motion and the ground truth, reflecting the quality of
the generated motion sequences. R-Precision assesses the alignment quality
between text and motion embeddings, reporting Top-1, Top-2, and Top-3
accuracy, with a batch size of 32 for calculation. Multimodal Distance (MM-
Dist) computes the average distance between each text and motion embedding
pair, while Multimodality (MModality) calculates the pairwise distance between
10 pairs of motion sequences generated from the same text description. Lastly,
for Diversity, we randomly sample 300 pairs of motion sequences from the
entire set and compute the average distance.

Table 2: Comparison with the state-of-the-art methods on our RICH-CAT test
set. * denotes that we replace the motion representation in the method with a
concatenated representation of motion and contact states. Red and blue
highlight the best and the second best results. Following [12], we conduct the
evaluation process 20 times and report the average results with a 95%
confidence level.

Methods | R-Precision $\uparrow$ | FID $\downarrow$ | MM-Dist $\downarrow$ | Diversity $\uparrow$ | MModality $\uparrow$
---|---|---|---|---|---
Top-1 | Top-2 | Top-3
Real motion | $0.492^{\pm 0.003}$ | $0.746^{\pm 0.002}$ | $0.870^{\pm 0.002}$ | $0.000^{\pm 0.000}$ | $3.020^{\pm 0.003}$ | $12.868^{\pm 0.111}$ | -
Motion VQVAE | $0.471^{\pm 0.003}$ | $0.722^{\pm 0.003}$ | $0.847^{\pm 0.003}$ | $0.443^{0.004}$ | $3.382^{\pm 0.004}$ | $12.933^{\pm 0.097}$ | -
MDM [42] | $0.360^{\pm 0.004}$ | $0.561^{\pm 0.001}$ | $0.673^{\pm 0.005}$ | $0.949^{\pm 0.037}$ | $4.873^{\pm 0.032}$ | $13.077^{\pm 0.085}$ | ${2.358}^{\pm 0.139}$
T2M-GPT [49] | $0.462^{\pm 0.004}$ | $0.697^{\pm 0.005}$ | $0.816^{\pm 0.003}$ | $1.195^{\pm 0.034}$ | $3.385^{\pm 0.018}$ | $13.089^{\pm 0.119}$ | ${2.396}^{\pm 0.043}$
MLD [3] | ${\color[rgb]{0,0,1}0.489}^{\pm 0.003}$ | ${\color[rgb]{0,0,1}0.741}^{\pm 0.002}$ | ${\color[rgb]{0,0,1}0.866}^{\pm 0.002}$ | ${\color[rgb]{0,0,1}0.771}^{\pm 0.012}$ | ${\color[rgb]{1,0,0}2.904}^{\pm 0.014}$ | ${\color[rgb]{1,0,0}13.396}^{\pm 0.101}$ | $2.095^{\pm 0.082}$
MDM* [42] | ${0.102}^{\pm{0.003}}$ | ${0.179}^{\pm{0.004}}$ | ${0.247}^{\pm{0.006}}$ | ${20.620}^{\pm{0.355}}$ | ${10.144}^{\pm{0.047}}$ | ${11.631}^{\pm{0.182}}$ | ${\color[rgb]{1,0,0}{6.589}}^{\pm{0.001}}$
T2M-GPT*[49] | ${0.466}^{\pm{0.003}}$ | ${0.696}^{\pm{0.004}}$ | ${0.808}^{\pm{0.004}}$ | ${0.895}^{\pm{0.025}}$ | ${3.512}^{\pm{0.019}}$ | ${12.784}^{\pm{0.110}}$ | ${2.607}^{\pm{0.095}}$
MLD*[3] | ${0.482}^{\pm{0.003}}$ | ${0.718}^{\pm{0.003}}$ | ${0.832}^{\pm{0.002}}$ | ${0.868}^{\pm{0.015}}$ | ${3.225}^{\pm{0.016}}$ | ${13.068}^{\pm{0.098}}$ | ${\color[rgb]{0,0,1}{2.622}}^{\pm{0.074}}$
Ours | ${\color[rgb]{1,0,0}0.512}^{\pm 0.003}$ | ${\color[rgb]{1,0,0}0.755}^{\pm 0.004}$ | ${\color[rgb]{1,0,0}0.870}^{\pm 0.003}$ | ${\color[rgb]{1,0,0}0.743}^{\pm 0.017}$ | ${\color[rgb]{0,0,1}2.906}^{\pm 0.012}$ | ${\color[rgb]{0,0,1}13.125}^{\pm 0.121}$ | $2.238^{\pm 0.049}$

###  5.2 Main Results

####  5.2.1 Quantitative results.

Table 2 displays the numerical results on our RICH-CAT test set. We compare
our model with three state-of-the-art methods and their modified versions:
T2M-GPT [49], MDM [42] and MLD [3]. The results demonstrate that our method
outperforms the baselines and their variants in both motion generation and
text-motion consistency. Particularly, our method outperforms others
significantly in the FID metric, indicating high-quality interactive motion
generation. In terms of text-motion consistency, we outperform other methods
significantly in R precision metrics and achieve comparable outcomes in
multimodal distance compared to MLD [3]. Additionally, compared with the
original versions, T2M-GPT* performs better with integrated contact maps,
while MLD* experiences a slight degradation. Notably, MDM* collapses, due to
the diffusion model’s limited suitability for handling motion and interaction
data with diverse distributions and data formats. MLD* avoids this issue by
leveraging VAE to extract powerful latent before employing diffusion model.

####  5.2.2 Qualitative results.

Figure 3 shows the visual results on our RICH-CAT test set. We compare our
results with T2M-GPT [49], MDM [42], and MLD [3]. The generation results in
Figure 3 show that our method can accurately understand all the factors in the
text, including action, interaction object (fence), and interaction body part
(left leg). The other methods exhibit mismatches between the action “climb
over the fence” and “stretch on the fence”, resulting in a wrong action type.
In contrast, our method can distinguish the two similar poses by the learned
contact data, and respond to the object specified in the textual input. As
depicted in Figure 3(c), MLD accurately identifies the action “stretches the
left leg”, but it lacks effective interaction with the object “fence”. The
left leg should remain stationary, yet MLD’s result displays noticeable
jittering in both upward and downward directions, suggesting an inaccurate
perception of the object’s position. In contrast, our approach maintains the
left leg in stable and realistic positions.

Figure 3: Qualitative comparison with the state-of-the-art methods on RICH-CAT
test set. We compare (f) our method with (b) MDM [42], (c) MLD [3], (e)
T2M-GPT [49]. Part of (a) ground-truth motion and (d) ground-truth contact is
provided for reference.

###  5.3 Ablation Study

####  5.3.1 Motion and contact modeling

. Table 3 illustrates the performance of different components of our method on
the RICH-CAT test set. The combination of our Motion VQ-VAE and Contact VQ-
VAE, even without using the intertwined GPT, proves highly effective compared
to a single Motion VQ-VAE, as evidenced by a significant decrease in FID. This
implies that integrating contact information can effectively enhance motion
quality. The introduction of the intertwined GPT and the multi-modal
corruption strategy results in a substantial improvement in R precision Top-1
accuracy and FID, showcasing an enhanced model ability in text-motion
retrieval. This improvement is attributed to the mutually conditioned
prediction of motion and contact data in the intertwined GPT module. Upon
further integration of the interaction-aware text encoder, there is a
remarkable enhancement in text-motion alignment, with an approximately 3.7%
increase in Top-1 accuracy. This is attributed to our text encoder’s enhanced
understanding of interactions in textual descriptions. The addition of the
alignment loss further improves the accuracy of motion generation while
maintaining a high level of text-motion alignment. Please refer to the
supplementary materials for complete results and detailed analysis.

Table 3: Ablation study. “Motion”: Motion VQ-VAE. “Contact”: Contact VQ-VAE.
“iGPT”: intertwined GPT. “Corrupt”: multi-modal corruption training strategy.
“$\mathcal{E}_{t}$”: interaction-aware text encoder pretrained on our RICH-CAT
training set. “Align”: text-motion alignment loss.

Motion | Contact | GPT | iGPT | Corrupt | $\mathcal{E}_{t}$ | Align | FID $\downarrow$ | Top-1 $\uparrow$
---|---|---|---|---|---|---|---|---
✓ |  | ✓ |  |  |  |  | $1.195^{\pm 0.034}$ | $0.462^{\pm 0.004}$
✓ | ✓ | ✓ |  |  |  |  | ${0.833}^{\pm{0.028}}$ | ${0.455}^{\pm{0.003}}$
✓ | ✓ |  | ✓ |  |  |  | $0.883^{\pm 0.022}$ | $0.476^{\pm 0.002}$
✓ | ✓ |  | ✓ | ✓ |  |  | $0.836^{\pm 0.029}$ | $0.478^{\pm 0.004}$
✓ | ✓ |  | ✓ | ✓ | ✓ |  | $0.772^{\pm 0.023}$ | $\mathbf{{0.515}}^{\pm{0.003}}$
✓ | ✓ |  | ✓ | ✓ | ✓ | ✓ | $\mathbf{{0.743}}^{\pm{0.017}}$ | ${0.512}^{\pm{0.003}}$

Table 4: Ablation study for motion and contact codebook sizes. “M”: the
codebook size of Motion VQ-VAE. “C”: the codebook size of Contact VQ-VAE. All
the models are trained without alignment loss.

M | C | FID $\downarrow$ | Top-1 $\uparrow$ | MM-Dist $\downarrow$ | M | C | FID $\downarrow$ | Top-1 $\uparrow$ | MM-Dist $\downarrow$
---|---|---|---|---|---|---|---|---|---
$256$ | $512$ | ${2.521}^{\pm{0.058}}$ | ${0.479}^{\pm{0.003}}$ | ${3.903}^{\pm{0.027}}$ | $512$ | $256$ | $\mathbf{{0.727}}^{\pm{0.018}}$ | $\mathbf{{0.517}}^{\pm{0.003}}$ | ${2.856}^{\pm{0.019}}$
$512$ | $512$ | ${0.772}^{\pm{0.023}}$ | $\mathbf{{0.515}}^{\pm{0.003}}$ | $\mathbf{{2.831}}^{\pm{0.016}}$ | $512$ | $512$ | ${0.772}^{\pm{0.023}}$ | ${0.515}^{\pm{0.003}}$ | $\mathbf{{2.831}}^{\pm{0.016}}$
$1024$ | $512$ | $\mathbf{{0.602}}^{\pm{0.022}}$ | ${0.511}^{\pm{0.004}}$ | ${2.968}^{\pm{0.022}}$ | $512$ | $1024$ | ${4.365}^{\pm{0.120}}$ | ${0.437}^{\pm{0.003}}$ | ${4.513}^{\pm{0.034}}$

Figure 4: Qualitative results of our method adapted to HOI synthesis.

####  5.3.2 Impact of codebook sizes.

We set one codebook size to a fixed value of $512$ and experiment with the
other, varying from $[256,512,1024]$. Table 4 shows that a larger codebook for
motion and a smaller one for contact lead to better outcomes. Please see
supplementary materials for more results and analysis.

###  5.4 Application to HOI Synthesis

While our method focuses on the generation of interactive human motions
conditioned on text, it can be readily extended to the task of Human-Object
Interaction (HOI) generation by incorporating an object geometry module. The
motion, aligned with the object can be predicted conditioned on both the
object feature and text embeddings. Results are shown in Figure 4. More
details are provided in the supplementary materials.

##  6 Conclusion

In this work, we introduce a novel task of generating 3D interactive human
motion from text. We construct a new dataset named RICH-CAT, derived from RICH
dataset [16], featuring high-quality motion data, precise human-object contact
labels, and interactive textual descriptions. Leveraging RICH-CAT, we present
CATMO, a novel approach for text-driven interactive human motion synthesis,
emphasizing body contact as key evidence. Our method involves the independent
modeling of motion and contact through VQ-VAEs, coupled with conditioned
generation via an intertwined GPT. Additionally, an interaction-aware text
encoder is employed to enhance the understanding of interactive textual
descriptions. Experiments showcase the superior performance of our method in
generating accurate and context-aware motion sequences from textual input. We
also demonstrate the adaptability of our model for HOI generation.

Contact-aware Human Motion Generation from Textual Descriptions: Supplementary
Material

We delve into the design and training specifics of our method in Section 7.
Additionally, Section 8 provides insights into the implementation details of
comparison methods, offers more visual results, presents comprehensive
experiment results, conducts a thorough analysis of the ablation study, and
explores the application of our method to HOI synthesis task. The discussion
on limitations and avenues for future work is presented in Section 9.
Furthermore, we discuss the risks and potential misuse of our proposed dataset
and model in Section 10. For a more in-depth look at the visual results,
please refer to the supplementary video demo.

Figure 5: Qualitative comparison with the state-of-the-art methods on RICH-CAT
dataset. We compare (g) our method with (b) T2M-GPT [49], (d) MDM [42], (e)
MLD [3]. Part of (a,c) ground-truth motion and (f) ground-truth contact is
provided for reference.

##  7 Method

###  7.1 Motion and Contact VQ-VAEs

####  7.1.1 Implementation details.

Motion VQ-VAE and Contact VQ-VAE both adopt a series of Conv1D and residual
blocks with a downsampling rate set to $l=4$. During training, we employ
Exponential Moving Average (EMA), codebook reset [44, 49] training strategies
to prevent codebook collapse. For both VQ-VAEs, the codebook size is set to
512. However, this doesn’t imply that Motion VQ-VAE and Contact VQ-VAE must
have the same codebook size. We conducted an ablation study on the codebook
sizes of both, and the details are presented in Section 8.4.

###  7.2 Interaction-aware text encoder

####  7.2.1 Architecture.

The interaction-aware text encoder is trained alongside a movement encoder,
aiming at generating text and motion embeddings that seamlessly align. Each
encoder comprises multiple convolutional and MLP layers. Specifically, the
interaction-aware text encoder is designed to encode detailed textual
descriptions, incorporating labels for interacting objects and body parts. On
the other hand, the movement encoder specializes in encoding motion sequences
into distinct latent spaces. Movement encoder consists of two parts: the first
part is trained with reconstruction loss to extract motion embeddings, while
the second part, comprising additional MLP layers, is trained with contrastive
loss to facilitate motion-text alignment.

The model undergoes pretraining using the training set of our RICH-CAT
dataset. The initial training phase involves independently training the first
part of the movement encoder with the reconstruction loss. Subsequently,
during joint training of the text encoder and movement encoder with
contrastive loss for alignment, the first part of the movement encoder remains
frozen.

####  7.2.2 Text encoder in intertwined GPT.

While training the intertwined GPT, the text encoder remains frozen and
produce text embeddings as input for the intertwined GPT. The alignment loss,
using the same contrastive loss, is employed between the text embedding and
the motion embedding to finetune the intertwined GPT. This motion embedding is
generated by the frozen movement encoder based on the prediction from the
intertwined GPT.

####  7.2.3 Optimization goal.

Following [14], we use the contrastive loss to align the text and motion
spaces. Given the motion embedding $Z$ and a text embedding $Z_{t}$, we can
calculate the pairwise euclidean distance between $Z$ and $Z_{t}$ as follows.

$D(Z_{t},Z)=||Z_{t}-Z||_{2}.$ (5)

Let $Y$ be a binary label to indicate whether the text and motion embeddings
belong to the same sequence. if text and motion embeddings belong to the same
distance, $Y=0$, otherwise, $Y=1$. Then the loss function can be formulated as

$\begin{split}&\mathcal{L}=(1-Y)\mathcal{L}_{s}(Z_{t},Z)+Y\mathcal{L}_{d}(Z_{t},Z),\\\
&\mathcal{L}_{s}=D(Z_{t},Z),\mathcal{L}_{d}=D(Z_{t},Z).\end{split}$ (6)

Where $\mathcal{L}_{s}$ is a positive loss function for a pair of text and
motion embeddings that belong to the same sequence, and $\mathcal{L}_{d}$ is
the negative loss function for the data that belongs to different sequences.
During implementation, we randomly shifted the data in a batch by an offset,
and calculate the negative loss between the text embedding $Z_{t}$ and the
randomly shifted motion embedding $Z^{\prime}$. A margin $m=10$ is adopted to
control the magnitude of negative loss as follows,

$\mathcal{L}_{d}=||max(0,m-D(Z_{t},Z^{\prime}))||_{2}.$ (7)

##  8 Experiment

###  8.1 Implementation details

We evaluate our method CATMO and three state-of-the-art text-to-motion methods
on our RICH-CAT dataset, including one latent-space method T2M-GPT [49] and
two diffusion-based methods MDM [42], MLD [3]. For a fair comparison, we
modified the above methods by directly concatenating the contact features with
the motion inputs, denoting MDM*, T2M-GPT*, MLD*. All the methods are trained
on our RICH-CAT training set and evaluated on the test set.

We implement comparison methods on our RICH-CAT dataset using their official
repositories. For the VQ-VAE models employed in both T2M-GPT [49] and our
approach, we set the codebook size and downsampling rate $l$ to $512$ and $4$,
respectively. We train 300,000 iterations for all VQ-VAEs and 100,000
iterations for GPT, using a learning rate of $2e^{-4}$ and a batch size of
$16$. For the diffusion models, MDM [42] and MLD [3], default settings are
adopted. For MLD [3], we conduct a training of 1500 epochs for the VAE and an
additional 1500 epochs for the diffusion model. For their variants, we replace
the motion input with the concatenation of both motion and contact features.

Table 5: Ablation study. “M”: Motion VQ-VAE. “C”: Contact VQ-VAE. “iGPT”:
intertwined GPT. “Cr”: multi-modal corruption training strategy.
“$\mathcal{E}_{t}$”: interaction-aware text encoder pretrained on our RICH-CAT
training set. “Align”: text-motion alignment loss.

M | C | GPT | iGPT | Cr | $\mathcal{E}_{t}$ | align | R-Precision $\uparrow$ | FID $\downarrow$ | MM-Dist $\downarrow$ | Diversity $\uparrow$ | MModality $\uparrow$
---|---|---|---|---|---|---|---|---|---|---|---
Top-1 | Top-2 | Top-3
✓ |  | ✓ |  |  |  |  | ${0.462}^{\pm{0.004}}$ | ${0.697}^{\pm{0.005}}$ | ${0.816}^{\pm{0.003}}$ | ${1.195}^{\pm{0.034}}$ | ${3.385}^{\pm{0.018}}$ | ${13.089}^{\pm{0.119}}$ | ${2.396}^{\pm{0.043}}$
✓ | ✓ | ✓ |  |  |  |  | ${0.455}^{\pm{0.003}}$ | ${0.690}^{\pm{0.003}}$ | ${0.814}^{\pm{0.002}}$ | ${0.833}^{\pm{0.028}}$ | ${3.497}^{\pm{0.014}}$ | ${13.039}^{\pm{0.098}}$ | ${2.708}^{\pm{0.111}}$
✓ | ✓ |  | ✓ |  |  |  | ${0.477}^{\pm{0.002}}$ | ${0.717}^{\pm{0.003}}$ | ${0.840}^{\pm{0.003}}$ | ${0.883}^{\pm{0.022}}$ | ${3.242}^{\pm{0.011}}$ | ${13.100}^{\pm{0.122}}$ | ${2.267}^{\pm{0.036}}$
✓ | ✓ |  | ✓ | ✓ |  |  | ${0.478}^{\pm{0.004}}$ | ${0.717}^{\pm{0.004}}$ | ${0.841}^{\pm{0.003}}$ | ${0.836}^{\pm{0.0029}}$ | ${3.227}^{\pm{0.001}}$ | ${13.071}^{\pm{0.088}}$ | ${2.211}^{\pm{0.069}}$
✓ | ✓ |  | ✓ | ✓ | ✓ |  | ${0.515}^{\pm{0.003}}$ | ${0.761}^{\pm{0.003}}$ | ${0.876}^{\pm{0.002}}$ | ${0.772}^{\pm{0.023}}$ | ${2.831}^{\pm{0.016}}$ | ${13.184}^{\pm{0.108}}$ | ${2.225}^{\pm{0.038}}$
✓ | ✓ |  | ✓ | ✓ | ✓ | ✓ | ${0.512}^{\pm{0.003}}$ | ${0.755}^{\pm{0.004}}$ | ${0.870}^{\pm{0.003}}$ | ${0.743}^{\pm{0.017}}$ | ${2.906}^{\pm{0.012}}$ | ${13.125}^{\pm{0.121}}$ | ${2.238}^{\pm{0.049}}$

Table 6: Ablation study for motion and contact codebook sizes. “M”: the
codebook size of Motion VQ-VAE. “C”: the codebook size of Contact VQ-VAE. All
the models are trained without alignment loss.

M | C | R-Precision $\uparrow$ | FID $\downarrow$ | MM-Dist $\downarrow$ | Diversity $\uparrow$ | MModality $\uparrow$
---|---|---|---|---|---|---
Top-1 | Top-2 | Top-3
$256$ | $512$ | ${0.479}^{\pm{0.003}}$ | ${0.700}^{\pm{0.003}}$ | ${0.804}^{\pm{0.003}}$ | ${2.521}^{\pm{0.058}}$ | ${3.903}^{\pm{0.027}}$ | ${13.171}^{\pm{0.079}}$ | ${3.736}^{\pm{0.125}}$
$512$ | $512$ | ${0.515}^{\pm{0.003}}$ | ${0.761}^{\pm{0.003}}$ | ${0.876}^{\pm{0.002}}$ | ${0.772}^{\pm{0.023}}$ | ${2.831}^{\pm{0.016}}$ | ${13.184}^{\pm{0.108}}$ | ${2.225}^{\pm{0.038}}$
$1024$ | $512$ | ${0.511}^{\pm{0.004}}$ | ${0.744}^{\pm{0.003}}$ | ${0.853}^{\pm{0.003}}$ | ${0.602}^{\pm{0.022}}$ | ${2.968}^{\pm{0.022}}$ | ${12.908}^{\pm{0.098}}$ | ${2.583}^{\pm{0.077}}$
$512$ | $256$ | ${0.517}^{\pm{0.003}}$ | ${0.762}^{\pm{0.003}}$ | ${0.875}^{\pm{0.003}}$ | ${0.727}^{\pm{0.018}}$ | ${2.856}^{\pm{0.019}}$ | ${13.268}^{\pm{0.104}}$ | ${2.122}^{\pm{0.100}}$
$512$ | $512$ | ${0.515}^{\pm{0.003}}$ | ${0.761}^{\pm{0.003}}$ | ${0.876}^{\pm{0.002}}$ | ${0.772}^{\pm{0.023}}$ | ${2.831}^{\pm{0.016}}$ | ${13.184}^{\pm{0.108}}$ | ${2.225}^{\pm{0.038}}$
$512$ | $1024$ | ${0.437}^{\pm{0.003}}$ | ${0.643}^{\pm{0.005}}$ | ${0.748}^{\pm{0.004}}$ | ${4.365}^{\pm{0.120}}$ | ${4.513}^{\pm{0.034}}$ | ${12.613}^{\pm{0.135}}$ | ${4.171}^{\pm{0.143}}$

###  8.2 Visual results

Figures 3 (in the main paper) and 5 showcase visual results on our RICH-CAT
dataset, comparing our method CATMO with T2M-GPT [49], MDM [42], and MLD [3].

In Figure 5, CATMO accurately executes the burpee jump action, maintaining
coherent ground contacts. T2M-GPT [49] and MDM [42] fail to complete the
action, with MDM [42] exhibiting significant jittering and implausible motion.
While MLD [3] performs the action correctly, it shows skating and jittering on
the feet. In contrast, CATMO completes the entire burpee jump action on the
ground with accurate body-ground interactions.

Figure 3, included in the main paper, is omitted here. For additional visual
results, please refer to the supplementary video demo

###  8.3 Ablation study: motion and contact modeling

Table 5 presents the performance analysis of various components within our
method on the RICH-CAT dataset. The combination of our Motion VQ-VAE and
Contact VQ-VAE, even without the use of the intertwined GPT, demonstrates
notable effectiveness compared to a singular Motion VQ-VAE, as evidenced by a
significant decrease in FID, approximately 0.36. The multimodality metric also
sees an improvement of about 0.31, indicating that integrating contact
information can significantly enhance motion quality.

The introduction of the intertwined GPT and multi-modal corruption strategy
results in a substantial enhancement in R precision, with an increase of
around 2.3% in Top-1 accuracy, 2.7% in Top-2 and Top-3 accuracy each,
showcasing an improved model ability in text-motion retrieval. The multimodal
distance also exhibits a noticeable improvement by 0.27, indicating a more
realistic motion prediction. This improvement is attributed to the mutually
conditioned prediction of motion and contact data in the intertwined GPT
module. Without employing a multimodal corruption strategy, the FID metric
shows degradation, while the performance of other metrics remains competitive.
Please note that models not utilizing a multi-modal corruption strategy employ
a standard corruption strategy during training.

Upon further integration of the interaction-aware text encoder, there is a
remarkable enhancement in text-motion alignment, with an increase of
approximately 3.7%, 4.4%, and 3.5% in Top-1, Top-2, and Top-3 accuracy,
respectively, and a decrease of about 0.06 in FID. This improvement is
attributed to our text encoder’s enhanced understanding of interactions in
textual descriptions.

The addition of the alignment loss further improves the accuracy of motion
generation, resulting in a decrease in FID by 0.03, while maintaining a
competitive text-motion alignment quality.

###  8.4 Ablation study: codebook sizes

To explore the impact of the codebook sizes of both Motion and Contact VQ-
VAEs, We set one codebook size to a fixed value of $512$ and experiment with
the other, varying from $[256,512,1024]$. We then compare the results of VQ-
VAEs and intertwined GPT trained with these parameters on RICH-CAT test set.

Analyzing the outcomes in Table 6, it becomes evident that employing a larger
codebook size for Motion VQ-VAE and a smaller codebook size for Contact VQ-VAE
yields superior results. When the contact codebook is fixed at 512, a small
motion codebook size (256) results in significantly poorer outcomes, including
a notable increase in FID, approximately 1.75, and a distinct decrease in R
precision, surpassing 3% in Top-3 accuracy. Similarly, setting the motion
codebook size to 512, in cases where the contact codebook size is large
(1024), leads to comparable negative effects. Intuitively, motion features,
with a higher dimensionality (263 dimensions), are more intricate compared to
contact features (22 dimensions), necessitating a larger codebook.

###  8.5 Ablation study: prediction order of intertwined GPT

The intertwined GPT is engineered to predict both the motion token and the
contact token in a mutually conditioned fashion. Initially, the model predicts
the motion token, followed by the contact token. To investigate the influence
of this prediction order, we have experimented with reversing it. The results
are summarized in Table 7, where it can be seen that even with the reversed
order, the intertwined GPT continues to achieve good results on our RICH-CAT
test set.

Table 7: Ablation study for prediction order of intertwined GPT.

Order | R-Precision $\uparrow$ | FID $\downarrow$ | MM-Dist $\downarrow$ | Diversity $\uparrow$ | MModality $\uparrow$
---|---|---|---|---|---
Top-1 | Top-2 | Top-3
motion $\rightarrow$ contact | ${0.512}^{\pm 0.003}$ | ${0.755}^{\pm 0.004}$ | ${0.870}^{\pm 0.003}$ | ${0.743}^{\pm 0.017}$ | ${2.906}^{\pm 0.012}$ | ${13.125}^{\pm 0.121}$ | $2.238^{\pm 0.049}$
contact $\rightarrow$ motion | ${0.514}^{\pm{0.003}}$ | ${0.760}^{\pm{0.003}}$ | ${0.876}^{\pm{0.002}}$ | ${0.711}^{\pm{0.015}}$ | ${2.863}^{\pm{0.011}}$ | ${13.306}^{\pm{0.133}}$ | ${2.204}^{\pm{0.054}}$

###  8.6 Application to HOI synthesis

Human-object interaction (HOI) generation [2, 47, 20, 7, 26] aims to produce a
holistic human and object motion sequence conditioned on the object’s
geometry, sparse object motion evidence, and optionally a textual description.
Different from HOI, our task focuses on modeling the alignment between textual
descriptions and human motion, particularly the varied interactions between
human body parts and objects as detailed in the text. Furthermore, our method
exhibits versatility in adapting to HOI generation in static scenes. Through
the integration of a conditioning module for object geometry, our approach
adeptly generates natural and physically plausible motion sequences that
interact with the given static objects. Visual results are presented in Figure
4 (in main paper) and 6. Please refer to the supplementary video demo for more
visual results.

####  8.6.1 Implementation details.

To handle the object geometry represented as an object mesh, we utilize the
Point Transformer [52], pretrained on ScanNet [6]. Initially, the object mesh
with N vertices is uniformly sampled down to a fixed length of 1024 points.
Subsequently, the Point Transformer produces a 512-dimensional feature vector.
This object feature, when concatenated with the text embeddings, serves as
input conditions for the intertwined GPT module. This setup enables the module
to predict realistic motion and contact tokens that align with both the object
and textual inputs.

Figure 6: Visual results of our method adapted to HOI synthesis task.

##  9 Limitation and Future Work

Although our method effectively generates realistic and diverse motion
sequences based on textual inputs, several limitations warrant attention in
future research. Firstly, accurately modeling delicate hand actions poses a
challenge. Despite the current model’s adept handling of whole-body contacts,
finer hand movements sometimes fail to precisely align with object geometries,
necessitating separate modeling of hand-action contacts. Secondly, the current
framework is tailored for static objects and lacks provisions for dynamic
objects that move alongside humans. The integration of these diverse object
types within a unified HOI framework presents a significant avenue for further
investigation. Lastly, despite the introduction of an efficient automatic text
annotation pipeline, minor disparities between annotations and natural
language may persist. Delving into the impact of these differences on the
model and exploring advanced techniques, such as leveraging large language
models, to elevate the quality of automatic annotation, stands as a promising
path for exploration.

##  10 Risks and Potential Misuse

Our focus in this study lies on exploring text-driven interactive motion
generation. Given the inclusion of realistic and diverse 3D human motion
sequences and videos within our dataset, coupled with our model’s capacity to
generate motion sequences, there exists a potential risk for misuse in the
creation of fake avatars and videos. Therefore, it is imperative that we
approach the use of this technology with responsibility and caution. Our aim
is to raise public awareness regarding the safe and ethical utilization of
such advancements.

## References

  * [1] Ahuja, C., Morency, L.P.: Language2pose: Natural language grounded pose forecasting. In: 2019 International Conference on 3D Vision (3DV). pp. 719–728. IEEE (2019)
  * [2] Bae, J., Won, J., Lim, D., Min, C.H., Kim, Y.M.: Pmp: Learning to physically interact with environments using part-wise motion priors. arXiv preprint arXiv:2305.03249 (2023)
  * [3] Chen, X., Jiang, B., Liu, W., Huang, Z., Fu, B., Chen, T., Yu, G.: Executing your commands via motion diffusion in latent space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18000–18010 (2023)
  * [4] Dabral, R., Mughal, M.H., Golyanik, V., Theobalt, C.: Mofusion: A framework for denoising-diffusion-based motion synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9760–9770 (2023)
  * [5] Dabral, R., Shimada, S., Jain, A., Theobalt, C., Golyanik, V.: Gravity-aware monocular 3d human-object reconstruction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12365–12374 (2021)
  * [6] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: Scannet: Richly-annotated 3d reconstructions of indoor scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5828–5839 (2017)
  * [7] Diller, C., Dai, A.: Cg-hoi: Contact-guided 3d human-object interaction generation. arXiv preprint arXiv:2311.16097 (2023)
  * [8] Gärtner, E., Andriluka, M., Coumans, E., Sminchisescu, C.: Differentiable dynamics for articulated 3d human motion reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13190–13200 (2022)
  * [9] Ghosh, A., Cheema, N., Oguz, C., Theobalt, C., Slusallek, P.: Synthesis of compositional animations from textual descriptions. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1396–1406 (2021)
  * [10] Ghosh, A., Dabral, R., Golyanik, V., Theobalt, C., Slusallek, P.: Imos: Intent-driven full-body motion synthesis for human-object interactions. In: Computer Graphics Forum. vol. 42, pp. 1–12. Wiley Online Library (2023)
  * [11] Gong, K., Li, B., Zhang, J., Wang, T., Huang, J., Mi, M.B., Feng, J., Wang, X.: Posetriplet: co-evolving 3d human pose estimation, imitation, and hallucination under self-supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11017–11027 (2022)
  * [12] Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., Cheng, L.: Generating diverse and natural 3d human motions from text. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5152–5161 (2022)
  * [13] Guo, C., Zuo, X., Wang, S., Zou, S., Sun, Q., Deng, A., Gong, M., Cheng, L.: Action2motion: Conditioned generation of 3d human motions. In: Proceedings of the 28th ACM International Conference on Multimedia. pp. 2021–2029 (2020)
  * [14] Hadsell, R., Chopra, S., LeCun, Y.: Dimensionality reduction by learning an invariant mapping. In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06). vol. 2, pp. 1735–1742. IEEE (2006)
  * [15] Huang, B., Pan, L., Yang, Y., Ju, J., Wang, Y.: Neural mocon: Neural motion control for physically plausible human motion capture. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6417–6426 (2022)
  * [16] Huang, C.H.P., Yi, H., Höschle, M., Safroshkin, M., Alexiadis, T., Polikovsky, S., Scharstein, D., Black, M.J.: Capturing and inferring dense full-body human-scene contact. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13274–13285 (2022)
  * [17] Ji, Y., Xu, F., Yang, Y., Shen, F., Shen, H.T., Zheng, W.S.: A large-scale rgb-d database for arbitrary-view human action recognition. In: Proceedings of the 26th ACM international Conference on Multimedia. pp. 1510–1518 (2018)
  * [18] Jiang, B., Chen, X., Liu, W., Yu, J., Yu, G., Chen, T.: Motiongpt: Human motion as a foreign language. arXiv preprint arXiv:2306.14795 (2023)
  * [19] Kim, J., Kim, J., Choi, S.: Flame: Free-form language-based motion synthesis & editing. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 8255–8263 (2023)
  * [20] Li, J., Wu, J., Liu, C.K.: Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG) 42(6), 1–11 (2023)
  * [21] Liu, J., Shahroudy, A., Perez, M., Wang, G., Duan, L.Y., Kot, A.C.: Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding. IEEE transactions on pattern analysis and machine intelligence 42(10), 2684–2701 (2019)
  * [22] Lu, S., Chen, L.H., Zeng, A., Lin, J., Zhang, R., Zhang, L., Shum, H.Y.: Humantomato: Text-aligned whole-body motion generation. arXiv preprint arXiv:2310.12978 (2023)
  * [23] Luo, Z., Hachiuma, R., Yuan, Y., Kitani, K.: Dynamics-regulated kinematic policy for egocentric pose estimation. Advances in Neural Information Processing Systems 34, 25019–25032 (2021)
  * [24] Ma, S., Cao, Q., Yi, H., Zhang, J., Tao, D.: Grammar: Ground-aware motion model for 3d human motion reconstruction. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 2817–2828 (2023)
  * [25] Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A.A., Tzionas, D., Black, M.J.: Expressive body capture: 3d hands, face, and body from a single image. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10975–10985 (2019)
  * [26] Peng, X., Xie, Y., Wu, Z., Jampani, V., Sun, D., Jiang, H.: Hoi-diff: Text-driven synthesis of 3d human-object interactions using diffusion models. arXiv preprint arXiv:2312.06553 (2023)
  * [27] Petrovich, M., Black, M.J., Varol, G.: Temos: Generating diverse human motions from textual descriptions. In: European Conference on Computer Vision. pp. 480–497. Springer (2022)
  * [28] Petrovich, M., Black, M.J., Varol, G.: Tmr: Text-to-motion retrieval using contrastive 3d human motion synthesis. arXiv preprint arXiv:2305.00976 (2023)
  * [29] Plappert, M., Mandery, C., Asfour, T.: The kit motion-language dataset. Big data 4(4), 236–252 (2016)
  * [30] Punnakkal, A.R., Chandrasekaran, A., Athanasiou, N., Quiros-Ramirez, A., Black, M.J.: Babel: Bodies, action and behavior with english labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 722–731 (2021)
  * [31] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [32] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2), 3 (2022)
  * [33] Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019)
  * [34] Rempe, D., Birdal, T., Hertzmann, A., Yang, J., Sridhar, S., Guibas, L.J.: Humor: 3d human motion model for robust pose estimation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11488–11499 (2021)
  * [35] Rempe, D., Guibas, L.J., Hertzmann, A., Russell, B., Villegas, R., Yang, J.: Contact and human dynamics from monocular video. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16. pp. 71–87. Springer (2020)
  * [36] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [37] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35, 36479–36494 (2022)
  * [38] Shimada, S., Golyanik, V., Xu, W., Pérez, P., Theobalt, C.: Neural monocular 3d human motion capture with physical awareness. ACM Transactions on Graphics (ToG) 40(4), 1–15 (2021)
  * [39] Shimada, S., Golyanik, V., Xu, W., Theobalt, C.: Physcap: Physically plausible monocular 3d motion capture in real time. ACM Transactions on Graphics (ToG) 39(6), 1–16 (2020)
  * [40] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: International conference on machine learning. pp. 2256–2265. PMLR (2015)
  * [41] Tevet, G., Gordon, B., Hertz, A., Bermano, A.H., Cohen-Or, D.: Motionclip: Exposing human motion generation to clip space. In: European Conference on Computer Vision. pp. 358–374. Springer (2022)
  * [42] Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-Or, D., Bermano, A.H.: Human motion diffusion model. arXiv preprint arXiv:2209.14916 (2022)
  * [43] Tripathi, S., Chatterjee, A., Passy, J.C., Yi, H., Tzionas, D., Black, M.J.: Deco: Dense estimation of 3d human-scene contact in the wild. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8001–8013 (2023)
  * [44] Van Den Oord, A., Vinyals, O., et al.: Neural discrete representation learning. Advances in neural information processing systems 30 (2017)
  * [45] Wang, Z., Chen, Y., Liu, T., Zhu, Y., Liang, W., Huang, S.: Humanise: Language-conditioned human motion generation in 3d scenes. Advances in Neural Information Processing Systems 35, 14959–14971 (2022)
  * [46] Xie, K., Wang, T., Iqbal, U., Guo, Y., Fidler, S., Shkurti, F.: Physics-based human motion estimation and synthesis from videos. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11532–11541 (2021)
  * [47] Xu, S., Li, Z., Wang, Y.X., Gui, L.Y.: Interdiff: Generating 3d human-object interactions with physics-informed diffusion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14928–14940 (2023)
  * [48] Yuan, Y., Wei, S.E., Simon, T., Kitani, K., Saragih, J.: Simpoe: Simulated character control for 3d human pose estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 7159–7169 (2021)
  * [49] Zhang, J., Zhang, Y., Cun, X., Huang, S., Zhang, Y., Zhao, H., Lu, H., Shen, X.: T2m-gpt: Generating human motion from textual descriptions with discrete representations. arXiv preprint arXiv:2301.06052 (2023)
  * [50] Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., Liu, Z.: Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001 (2022)
  * [51] Zhang, S., Zhang, Y., Bogo, F., Pollefeys, M., Tang, S.: Learning motion priors for 4d human body capture in 3d scenes. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11343–11353 (2021)
  * [52] Zhao, H., Jiang, L., Jia, J., Torr, P.H., Koltun, V.: Point transformer. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 16259–16268 (2021)