# Executing your Commands via Motion Diffusion in Latent Space

Xin Chen1 Biao Jiang2 Wen Liu1 Zilong Huang1 Bin Fu1
Tao Chen2 Jingyi Yu3 Gang Yu1
1Tencent PCG 2Fudan University 3ShanghaiTech University
https://github.com/chenfengye/motion-latent-diffusion Corresponding author.

###### Abstract

We study a challenging task, conditional human motion generation, which
produces plausible human motion sequences according to various conditional
inputs, such as action classes or textual descriptors. Since human motions are
highly diverse and have a property of quite different distribution from
conditional modalities, such as textual descriptors in natural languages, it
is hard to learn a probabilistic mapping from the desired conditional modality
to the human motion sequences. Besides, the raw motion data from the motion
capture system might be redundant in sequences and contain noises; directly
modeling the joint distribution over the raw motion sequences and conditional
modalities would need a heavy computational overhead and might result in
artifacts introduced by the captured noises. To learn a better representation
of the various human motion sequences, we first design a powerful Variational
AutoEncoder (VAE) and arrive at a representative and low-dimensional latent
code for a human motion sequence. Then, instead of using a diffusion model to
establish the connections between the raw motion sequences and the conditional
inputs, we perform a diffusion process on the motion latent space. Our
proposed Motion Latent-based Diffusion model (MLD) could produce vivid motion
sequences conforming to the given conditional inputs and substantially reduce
the computational overhead in both the training and inference stages.
Extensive experiments on various human motion generation tasks demonstrate
that our MLD achieves significant improvements over the state-of-the-art
methods among extensive human motion generation tasks, with two orders of
magnitude faster than previous diffusion models on raw motion sequences.

Figure 1: Our Motion Latent-based Diffusion (MLD) model can achieve high-
quality and diverse motion generation given a text prompt. The darker colors
indicate the later in time, and the colored words refer to the motions with
same colored trajectory.

##  1 Introduction

Human motion synthesis has recently rapidly developed in a multi-modal
generative fashion. Various condition inputs, such as music [34, 33, 32],
control signals [45, 66, 65], action categories [46, 19], and natural language
descriptions [16, 47, 68, 18, 2, 28], provide a more convenient and human-
friendly way to animate virtual characters or even control humanoid robots. It
will benefit numerous applications in the game industry, film production,
VR/AR, and robotic assistance.

Among all conditional modalities, text-based conditional human motion
synthesis has been driving and dominating research frontiers because the
language descriptors provide a convenient and natural user interface for
people to interact with computers [47, 2, 74, 68, 28]. However, since the
distributions between the natural language descriptors and motion sequences
are quite different, it is not easy to learn a probabilistic mapping function
from the textural descriptors to the motion sequences. Two typical methods
address this problem: 1) the cross-modal compatible latent space between
motion and language [47, 2] and 2) the conditional diffusion model [74, 68,
28]. The formers, such as TEMOS [47], usually separately learn a motion
Variational AutoEncoder (VAE) and a text Variational Encoding (without
decoder) and then constrain the text encoder and the motion encoder into a
compatible latent space via the Kullback-Leibler (KL) divergences loss, which
pushes a foundational step forward on creating human motion sequences by
natural language inputs. However, since the distributions of natural languages
and motion sequences are highly different, forcibly aligning these two simple
gaussian distributions, in terms of variational text encoding and variational
motion encoding, into a compatible distribution might result in misalignments
and thereby reduce the generative diversity inevitably. In light of the
tremendous success of the diffusion-based generative models on other domains
[53, 61, 56, 22, 78, 72], the latter category methods [74, 68, 28] propose a
conditional diffusion model for human motion synthesis to learn a more
powerful probabilistic mapping from the textual descriptors to human motion
sequences and improve the synthesized quality and diversity. Nevertheless, the
raw motion sequences are somewhat time-axis redundant, and diffusion models in
raw sequential data [54, 22, 35] usually require exhausting computational
overhead in both the training and inference phase, which is inefficient.
Besides, since the raw motion data from the motion capture system might
contain noises, the powerful diffusion models might learn the clues of a
probabilistic mapping from the conditional inputs to the noise motion
sequences and produce artifacts.

To efficiently synthesize plausible and diverse human motion sequences
according to the conditional inputs, inspired by the success of the diffusion
model on latent space in text-to-image synthesis [56], we combine the
advantages of the latent space-based and the conditional diffusion-based
methods and propose a motion latent-based diffusion model (MLD) for human
motion generation. Specifically, we first design a transformer-based
autoencoder [46] with the UNet-like long skip connections [59] to learn a
representative and low-dimensional latent distribution of human motion
sequences. Then, instead of using a diffusion model to establish the
connections between the raw motion sequences and the conditional inputs, we
propose a motion latent-based diffusion model (MLD) to learn a better
probabilistic mapping from the conditions to the representative motion latent
codes, which could not only produce the vivid motion sequences conforming to
the given conditional inputs but also substantially reduce the computational
overhead in both training and inference stage. In addition, high-quality human
motion sequences with well-annotated action labels or textual descriptions are
expensive and limited. In contrast, the large-scale non-annotated or weakly
annotated motion sequences are publicly available, such as the AMASS dataset
[41]. Our proposed MLD could individually train a motion latent autoencoder on
these large-scale datasets, arriving at a representative and low-dimensional
latent space for diverse human motion sequences. This low-dimensional latent
space with higher information density could accelerate the model’s convergence
and significantly reduce computational consumption for the downstream
conditional human motion generation tasks.

We summarize the contributions as follows: 1) we design and explore a more
representative motion variational autoencoder (VAE), which provides state-of-
the-art motion reconstruction and diverse generation, benefiting the training
of the latent diffusion models; 2) we further demonstrate that motion
generation tasks on latent spaces, such as text-to-motion and action-to-
motion, are more efficient than the diffusion models on raw motion sequences;
3) our proposed MLD achieves competitive performance on multiple tasks
(unconditional motion generation, action-to-motion, and text-to-motion), and
codes are available.

##  2 Related Work

Human Motion Synthesis allows rich inputs of multi-modal data, such as text
[16, 47, 68, 18, 2, 28], action category [46, 19], incomplete pose sequences
[12, 20, 68], control signals [65, 66, 45], musics [34, 33, 32] and image(s)
[55, 9], here, we focus on some typical tasks. Firstly, unconditional motion
generation [73, 77, 75, 51, 68] is a more universal task, which models the
entire motion space, only needs motion data without any requirement of
annotation, and benefits other generation tasks. VPoser [43] proposes a
variational human pose prior mainly for image-based pose fitting. ACTOR [46,
47] recently proposes a class-agnostic transformer VAE as one baseline. After
that, among all conditional tasks, text-to-motion [47, 2, 74, 68, 28, 16] has
been driving and dominating research frontiers because the language
descriptors are the most user-friendly and convenient. More recently, two
categories of motion synthesis methods have emerged, joint-latent models [47,
2] and diffusion models [74, 68, 28]. The former category, like TEMOS [47],
proposes a VAE architecture to learn a joint latent space of motion and text
constrained on a Gaussian distribution. However, natural language and human
motions are quite different with misaligned structure and distribution, thus
it is difficult to forcibly align two simple Gaussian distributions. Lastly,
we introduce action-to-motion [46, 19], a reverse problem of the classical
action recognition task. ACTOR [46] proposes learnable biases in transformer
VAE to embed action for motion generation. However, most above methods can
only handle one task and hardly change condition inputs. We address this
problem by separating models into a universal motion generative model and
latent diffusion models to handle different motion generation tasks.

Motion data is critical in the development of motion synthesis tasks. Thanks
to the marker-based and markless motion capture approaches [31, 71, 21, 8],
they provide convenient and effective solutions for large raw motion data
collection. KIT Motion-Language [48] annotates sequence-level description for
motions from [42], and HumanML3d [17] provide more textual annotation for some
motions of AMASS[41]. They are also our focus in the text-to-motion task. For
the action-to-motion datasets, Babel [50] also collects motions from AMASS and
provides action and behavior annotations. ACTOR [46] use [31] to process two
action recognition datasets, HumanAct12 [19] and UESTC [26], for action-to-
motion task.

Motion Representation. These datasets lead to the discussion about motion
representation, such as the straightforward joint positions and the Master
Motor Map (MMM) format [67]. For our setting, we employ two motion
representations: 1) the classical SMPL-based [40, 31, 8] motion parameters and
2) the redundant hand-crafted motion feature [17, 66, 65] with a combination
of joints features. The former is widely used in motion capture, and the
latter is mainly used in character animation. As suggested by [17], we use the
latter in most of our synthesis framework to avoid foot-sliding issues, and
use the SMPL parameters for the action-based tasks for a fair comparison with
other approaches. Besides, we also recognize the latent in Tab. 4 as one of
motion representation.

Generative Models play an important role in motion synthesis tasks to generate
high-quality human motion, Although motion generative models, like VAEs [46,
39, 19] and Generative Adversarial Networks (GAN) [38, 1], can enable
effective human motion sampling, recent studies [3, 15, 19, 46] recommend VAEs
rather than GANs since the latter are more difficult to train. We follow their
suggestions and employ VAEs to compress and reconstruct human motion for the
learning of diffusion models. We next introduce the diffusion models,
especially in motion domain.

Diffusion Generative Models. Diffusion Generative Models [63] achieve
significant success in the image synthesis domain, such as Imagen [61], Dall2
[53] and Stable Diffusion [56]. Inspired by their works, most recent methods
[68, 74, 28] leverage diffusion models for human motion synthesis.
MotionDiffuse [68] is the first text-based motion diffusion model with fine-
grained instructions on body parts. MDM [68], most recently, proposes a motion
diffusion model on raw motion data to learn the relation between motion and
input conditions. However, these diffusion models are not very applicable to
raw motion data with potential noise and temporal consistency redundancy and
thus are easily misdirected by outliers. In addition, directly applying the
diffusion model [68, 74] to the raw motion data suffers from high
computational overheads and low inference speed. Inspired by [56], we propose
a motion latent-based diffusion model to reduce computational resources and
improve the generative quality.

##  3 Method

To efficiently generate high-quality and diverse human motion sequences
according to desired conditional inputs with fewer computational overheads, we
propose to perform a diffusion process on a representative and low-dimensional
motion latent space and consequently arrive at a motion latent-based diffusion
model (MLD) for conditional human motion synthesis. It contains a motion
Variational AutoEncoder (VAE) to learn a representative and low-dimensional
latent space for diverse human motion sequences (details in Sec. 3.1) and a
conditional diffusion model in this latent space (details in Sec. 3.2 and Sec.
3.3).

The conditions include action labels, textual descriptions, or even empty
conditions. Specifically, given an input condition $c$, such as a sentence
$\boldsymbol{w}^{1:N}=\\{w^{i}\\}_{i=1}^{N}$ describing a motion [47], a
action label $a$ from the predefined action categories set $a\in A$ [46] or
even a empty condition $c=\varnothing$ [43, 76], our MLD aims to generate a
human motion $\hat{x}^{1:L}=\\{\hat{x}^{i}\\}_{i=1}^{L}$ in a non-
deterministic way, where L denotes the motion length or frame number. Here, we
employ the motion representation in [17]: a combination of 3D joint rotations,
positions, velocities, and foot contact. In addition, we propose the motion
encoder $\mathcal{E}$ to encode the motion sequences,
${x}^{1:L}=\\{{x}^{i}\\}_{i=1}^{L}$, into a latent $z=\mathcal{E}(x^{1:L})$,
and decode $z$ into the motion sequences using a motion decoder $\mathcal{D}$,
that is $\hat{x}^{1:L}=\mathcal{D}(z)=\mathcal{D}(\mathcal{E}(x^{1:L}))$.

###  3.1 Motion Representation in Latents

We build our motion Variational AutoEncoder, $\mathcal{V}$, based on a
transformer-based architecture [46], which consists of a transformer encoder
$\mathcal{E}$ and a transformer decoder $\mathcal{D}$. The motion VAE,
$\mathcal{V}=\\{\mathcal{E},\mathcal{D}\\}$, is trained by the motion
$x^{1:L}$ reconstruction only with the Mean Squared Error (MSE) loss and the
Kullback-Leibler (KL) loss. We further enhance two transformers [69] of
$\mathcal{E}$ and $\mathcal{D}$ with long skip connections [59], and remove
the action biases used in [46]. The encoder could produce a representative,
low-dimensional latent space with high informative density, and the decoder
could well reconstruct the latent into motion sequences.

More specifically, the motion encoder $\mathcal{E}$ takes as input learnable
distribution tokens, and frame-wise motion features $x^{1:L}$ of arbitrary
length $L$. We use the embedded distribution tokens as Gaussian distribution
parameters $\mu$ and $\sigma$ of the motion latent space $\mathcal{Z}$ to
reparameterize [30] latent $z\in\mathbb{R}^{n\times d}$ whose dimension is
similar to [46]. The motion decoder $\mathcal{D}$ relies on the architecture
of the transformer decoder with cross attention mechanism, which takes the $L$
number of zero motion tokes as queries, a latent $z\in\mathbb{R}^{n\times d}$
as memory, and finally, generates a human motion sequence $\hat{x}^{1:L}$ with
$L$ frames.

According to [47], both the latent space $\mathcal{Z}$ and variable durations
help the model to produce more diverse motions. To further enhance the latent
representation, we leverage a long skip-connection structure for the
transformer-based encoder $\mathcal{E}$ and decoder $\mathcal{D}$. We also
explore the effectiveness of the latent’s dimensions on motion sequences
representation in Tab. 4. Hence, our VAE models present a stronger motion
reconstruction ability and richer diversity ($cf.$ Tab. 5 and Tab. 6). We
provide more details about the architecture and the training in the
supplementary.

###  3.2 Motion Latent Diffusion Model

Diffusion probabilistic models [63] can gradually anneal the noise from a
gaussian distribution to a data distribution $p(x)$ by learning the noise
prediction from a $T$-length Markov noising process, giving
$\\{\boldsymbol{x_{t}}\\}^{T}_{t=1}$. It leads to a significant influence in
many research domains, such as the most famous image synthesis models [11, 23,
62, 56], the density estimation model [29] and the motion generation models
[68, 74]. For motion generation, these works train the diffusion models with a
transformer-based denoiser $\epsilon_{\theta}\left(x_{t},t\right)$, which
anneal the random noise to motion sequence
$\\{\boldsymbol{\hat{x}_{t}^{1:N}}\\}^{T}_{t=1}$ iteratively.

However, diffusion on raw motion sequences is inefficient and requires
exhausting computational resources. Besides, raw motion data from the markless
or marker-based motion capture system usually remain high-frequency outliers,
which might have a side effect on the diffusion model to learn the actual data
distribution. To reduce the computational requirements of the diffusion models
on raw motion sequences and improve the synthesized quality, we perform the
diffusion process on a representative and low-dimensional motion latent space.

Here, we introduce our denoiser $\epsilon_{\theta}$. Different from the
previous UNet-based architecture [59] on the 2D image latent $z_{I}$, we build
a transformer-based denoising model with long skip connections [4] on the
motion latent $z\in\mathbb{R}^{n\times d}$, which is more suitable for
sequential data, like human motion sequences. The diffusion on latent space is
modeled as a Markov nosing process using:

$q\left(z_{t}\mid
z_{t-1}\right)=\mathcal{N}\left(\sqrt{\alpha_{t}}z_{t-1},\left(1-\alpha_{t}\right)I\right).$
(1)

where the constant $\alpha_{t}\in(0,1)$ is a hyper-parameters for sampling. We
then use $\\{z_{t}\\}^{T}_{t=0}$ to denote the noising sequence, and
$z_{t-1}=\epsilon_{\theta}\left(z_{t},t\right)$ for the $t$-step denoising. We
further focus on the unconditional generation with the simple objective [23]:

$L_{\text{MLD}}:=\mathbb{E}_{\epsilon,t}\left[\left\|\epsilon-\epsilon_{\theta}\left(z_{t},t\right)\right\|_{2}^{2}\right],$
(2)

where $\epsilon\sim\mathcal{N}(0,1)$, $z_{0}=\mathcal{E}(x^{1:L})$. During the
training of $\epsilon_{\theta}$, the encoder $\mathcal{E}$ is frozen to
compress motion into $z_{0}$. The samples of the diffusion forward process are
from the latent distribution $p(z_{0})$. During the diffusion reserve stage,
$\epsilon_{\theta}$ first predict $\hat{z}_{0}$ with $T$ iterative denoising
steps, then $\mathcal{D}$ decodes $\hat{z}_{0}$ to motion results with one
forward.

Figure 2: Method overview: MLD consists of a VAE model $\mathcal{V}$ (Sec.
3.1) and a latent diffusion model $\epsilon_{\theta}$ (Sec. 3.2) conditioned
on text or action embedding $\tau_{\theta}$ (Sec. 3.3). We propose two-stage
training: first learn $\mathcal{V}$ for Motion Representations in Latents and
then learn a conditioned denoiser $\epsilon_{\theta}$ from the diffusion
process $q\left(z_{t}\mid z_{t-1}\right)$. During inference, In practice, the
latent diffusion models predict the latent $\hat{z}_{0}$ from condition inputs
and then $\mathcal{D}$ decode it to motions efficiently.

###  3.3 Conditional Motion Latent Diffusion Model

Like many other diffusion models [68, 56, 61], our MLD model is also capable
of conditional motion generation $\mathcal{G}(c)$ by applying the conditional
distribution of $p(z|c)$, such as text [47, 16] and action [46, 19].
$\mathcal{G}(c)$ is implemented with conditional denoiser
$\epsilon_{\theta}(z_{t},t,c)$, which can share a common motion VAE model.
Therefore, for different conditions, only the learning of
$\epsilon_{\theta}(z_{t},t,c)$ is necessary. To address various $c$, the
domain encoder $\tau_{\theta}(c)\in\mathbb{R}^{m\times d}$ for condition
embedding benefits the denoiser $\epsilon_{\theta}(z_{t},t,\tau_{\theta}(c))$.

Figure 3: Qualitative comparison of the state-of-the-art methods. We provide
the visualized motion results and real references from three text prompts.
Under the same training and inference setting on HumanML3D [17], we find that
our generations better match the descriptions, but others have downgraded
motions or improper semantics.

Here we introduce two specific generation tasks, text-to-motion
$\mathcal{G}_{w}:w^{1:N}\mapsto x^{1:L}$ and action-to-motion
$\mathcal{G}_{a}:a\mapsto x^{1:L}$. Through investigation, CLIP [52] text
encoder $\tau_{\theta}^{w}(w^{1:N})\in\mathbb{R}^{1\times d}$ is employed to
map text prompt. On the other side, we build the learnable embedding for each
action category, giving $\tau_{\theta}^{a}(a)\in\mathbb{R}^{1\times d}$.
Injecting these embedded conditions into a transformer-based
$\epsilon_{\theta}$, two effective ways are concatenation and cross-attention,
and we figure out the former one seems to be more effective ($cf.$ Sec. 5 and
[68]) for motion diffusion models. Thus the conditional objective follows:

$L_{\text{MLD}}:=\mathbb{E}_{\epsilon,t,c}\left[\left\|\epsilon-\epsilon_{\theta}\left(z_{t},t,\tau_{\theta}(c)\right)\right\|_{2}^{2}\right].$
(3)

We freeze $\tau_{\theta}^{w}$ as suggested by [47] and joint optimize the
$\tau_{\theta}^{a}$ and $\epsilon_{\theta}$ via this objective. In addition,
our denoiser $\epsilon_{\theta}$ is learned with classifier-free diffusion
guidance [24], which is a trade-off to boost sample quality by reducing
diversity in conditional diffusion models. Specifically, it learns both the
conditioned and the unconditioned distribution with 10% dropout [61] of the
samples, and we perform a linear combination to in as followed:

$\epsilon^{s}_{\theta}(z_{t},t,c)=s(\epsilon_{\theta}(z_{t},t,c)+(1-s)\epsilon_{\theta}(z_{t},t,\varnothing)$
(4)

Here, $s$ is the guidance scale and $s>1$ can strengthen the effect of
guidance. After the interactive reverse process of the conditional denoising,
$\mathcal{D}$ reconstructs the motion from the predicted $\hat{z}_{0}$
efficiently.

##  4 Experiments

We provide extensive comparisons to evaluate our models on both quality and
efficiency in the following. Firstly, we introduce the datasets settings,
evaluation metrics and implementation details (Sec. 4.2). Importantly, we show
the comparisons on multiple datasets for different motion generation tasks
respectively, including text-to-motion (Sec. 4.3), action-to-motion (Sec. 4.4)
and unconditional generation (Sec. 4.5). More qualitative results, user
studies, and details are provided in supplements.

###  4.1 Datasets and Evaluation Metrics

Conditional motion synthesis can support rich inputs of multi-modal data, and
thus multiple datasets are utilized to evaluate MLD. We briefly introduce
these datasets. First is two text-to-motion datasets, HumanML3D and KIT [48],
and the latter provides 6,353 textual descriptions for 3,911 motions.
HumanML3D [17], a recent dataset, collects 14,616 motion sequences from AMASS
[41] and annotates 44,970 sequence-level textual description. We use its
motions, part of the AMASS, to evaluate unconditional task. As suggested by
[17], we use the redundant motion representation in a combination of joint
velocities, positions and rotations which is also used in [68, 74]. Lastly,
action-to-motion task requires action-conditioned motions similar to action
recognition datasets. Thanks to [46], after the processing, HumanAct12 [19]
provides 1,191 raw motion sequences and 12 action categories, and UESTC [26]
provides 24K sequences and 40 action categories. We rely on these two datasets
for action-to-motion evaluation.

Evaluation Metrics summarize in four parts. (a) Motion quality: Frechet
Inception Distance (FID) is our principal metric to evaluate the feature
distributions between the generated and real motions by feature extractor
[16]. To evaluate reconstruction error of VAEs, we use popular metrics in
motion capture [31, 8, 70], MPJPE and PAMPJPE [14] for global/local errors in
millimeters, Acceleration Error (ACCL) for temporal quality. (2) Generation
diversity: Diversity (DIV) calculates variance through features [16], while
MultiModality (MM) measures the generation diversity within the same text or
action input. (3) Condition matching: Under feature [16] space, motion-
retrieval precision (R Precision) calculates the text and motion Top 1/2/3
matching accuracy, and Multi-modal Distance (MM Dist) calculates the distance
between motions and texts. For action-to-motion, we use the corresponding
action recognition model [19][46] to calculate Accuracy (ACC) for action
categories. (4) Time costs: we propose Average Inference Time per Sentence
(AITS) measured in seconds to evaluate inference efficiency of diffusion
models. ($cf.$ supplements)

###  4.2 Implementation Details

For the comparisons, motion transformer encoders $\mathcal{E}$ and decoders
$\mathcal{D}$ of our VAE model $\mathcal{V}$ all consist of 9 layers and 4
heads with skip connection by default, as well as the transformer-based
denoiser $\epsilon_{\theta}$ in Sec. 3.2. The condition embedding
$\tau_{\theta}(c)\in\mathbb{R}^{1,256}$ and the latent
$z\in\mathbb{R}^{1,256}$ are concatenated for diffusion learning and
inference. We employ a frozen CLIP-ViT-L-14 model as the text encoder
$\tau_{\theta}^{w}$ for text condition, and a learnable embedding for action
condition. We leave the ablation on the components in Sec. 5, like the shape
of latent, the number of layers, injection of $z_{t}$ through the cross-
attention, and others. All our models are trained with the AdamW optimizer
using a fixed learning rate of $10^{-4}$. Our mini-batch size is set to 128
during the VAE training stage and 64 during the diffusion training stage
separately. Each model was trained for 6K epochs during VAE stage and 3K
epochs during diffusion stage. The number of diffusion steps is 1K during
training while 50 during interfering, and the variances $\beta_{t}$ are scaled
linearly from $8.5\times 10^{-4}$ to 0.012. For runtime, training tasks 8
hours for VAEs $\mathcal{V}$ and 4 hours for denoiser $\epsilon_{\theta}$ on 8
Tesla V100 GPUs, and we test MLD with a single V100 in Fig. 6, but it also can
run inference on a general computer graphics card, such as RTX 2080/3060.

Methods | R Precision $\uparrow$ | FID$\downarrow$ | MM Dist$\downarrow$ | Diversity$\rightarrow$ | MModality$\uparrow$
---|---|---|---|---|---
Top 1 | Top 2 | Top 3
Real | $0.511^{\pm.003}$ | $0.703^{\pm.003}$ | $0.797^{\pm.002}$ | $0.002^{\pm.000}$ | $2.974^{\pm.008}$ | $9.503^{\pm.065}$ | -
Seq2Seq [49] | $0.180^{\pm.002}$ | $0.300^{\pm.002}$ | $0.396^{\pm.002}$ | $11.75^{\pm.035}$ | $5.529^{\pm.007}$ | $6.223^{\pm.061}$ | -
LJ2P [2] | $0.246^{\pm.001}$ | $0.387^{\pm.002}$ | $0.486^{\pm.002}$ | $11.02^{\pm.046}$ | $5.296^{\pm.008}$ | $7.676^{\pm.058}$ | -
T2G[5] | $0.165^{\pm.001}$ | $0.267^{\pm.002}$ | $0.345^{\pm.002}$ | $7.664^{\pm.030}$ | $6.030^{\pm.008}$ | $6.409^{\pm.071}$ | -
Hier [13] | $0.301^{\pm.002}$ | $0.425^{\pm.002}$ | $0.552^{\pm.004}$ | $6.532^{\pm.024}$ | $5.012^{\pm.018}$ | $8.332^{\pm.042}$ | -
TEMOS [47] | $0.424^{\pm.002}$ | $0.612^{\pm.002}$ | $0.722^{\pm.002}$ | $3.734^{\pm.028}$ | $3.703^{\pm.008}$ | $8.973^{\pm.071}$ | $0.368^{\pm.018}$
T2M [16] | $0.457^{\pm.002}$ | $0.639^{\pm.003}$ | $0.740^{\pm.003}$ | $1.067^{\pm.002}$ | $3.340^{\pm.008}$ | $9.188^{\pm.002}$ | $2.090^{\pm.083}$
MDM [68] | $0.320^{\pm.005}$ | $0.498^{\pm.004}$ | $0.611^{\pm.007}$ | $0.544^{\pm.044}$ | $5.566^{\pm.027}$ | $\boldsymbol{9.559}^{\pm.086}$ | $\boldsymbol{2.799}^{\pm.072}$
MLD (Ours) | $\boldsymbol{0.481}^{\pm.003}$ | $\boldsymbol{0.673}^{\pm.003}$ | $\boldsymbol{0.772}^{\pm.002}$ | $\boldsymbol{0.473}^{\pm.013}$ | $\boldsymbol{3.196}^{\pm.010}$ | $9.724^{\pm.082}$ | $2.413^{\pm.079}$

Table 1: Comparison of text-conditional motion synthesis on HumanML3D [17]
dataset. These metrics are evaluated by the motion encoder from [16]. Empty
MModality indicates the non-diverse generation methods. We employ real motion
as a reference and sort all methods by descending FIDs. The right arrow
$\rightarrow$ means the closer to real motion the better.

Methods | R Precision $\uparrow$ | FID$\downarrow$ | MM Dist$\downarrow$ | Diversity$\rightarrow$ | MModality$\uparrow$
---|---|---|---|---|---
Top 1 | Top 2 | Top 3
Real | $0.424^{\pm.005}$ | $0.649^{\pm.006}$ | $0.779^{\pm.006}$ | $0.031^{\pm.004}$ | $2.788^{\pm.012}$ | $11.08^{\pm.097}$ | -
Seq2Seq[49] | $0.103^{\pm.003}$ | $0.178^{\pm.005}$ | $0.241^{\pm.006}$ | $24.86^{\pm.348}$ | $7.960^{\pm.031}$ | $6.744^{\pm.106}$ | -
T2G[5] | $0.156^{\pm.004}$ | $0.255^{\pm.004}$ | $0.338^{\pm.005}$ | $12.12^{\pm.183}$ | $6.964^{\pm.029}$ | $9.334^{\pm.079}$ | -
LJ2P [2] | $0.221^{\pm.005}$ | $0.373^{\pm.004}$ | $0.483^{\pm.005}$ | $6.545^{\pm.072}$ | $5.147^{\pm.030}$ | $9.073^{\pm.100}$ | -
Hier [13] | $0.255^{\pm.006}$ | $0.432^{\pm.007}$ | $0.531^{\pm.007}$ | $5.203^{\pm.107}$ | $4.986^{\pm.027}$ | $9.563^{\pm.072}$ | $2.090^{\pm.083}$
TEMOS [47] | $0.353^{\pm.006}$ | $0.561^{\pm.007}$ | $0.687^{\pm.005}$ | $3.717^{\pm.051}$ | $3.417^{\pm.019}$ | $10.84^{\pm.100}$ | $0.532^{\pm.034}$
T2M [16] | $0.370^{\pm.005}$ | $0.569^{\pm.007}$ | $0.693^{\pm.007}$ | $2.770^{\pm.109}$ | $3.401^{\pm.008}$ | $\boldsymbol{10.91}^{\pm.119}$ | $1.482^{\pm.065}$
MDM [68] | $0.164^{\pm.004}$ | $0.291^{\pm.004}$ | $0.396^{\pm.004}$ | $0.497^{\pm.021}$ | $9.191^{\pm.022}$ | $10.85^{\pm.109}$ | $1.907^{\pm.214}$
MLD (Ours) | $\boldsymbol{0.390}^{\pm.008}$ | $\boldsymbol{0.609}^{\pm.008}$ | $\boldsymbol{0.734}^{\pm.007}$ | $\boldsymbol{0.404}^{\pm.027}$ | $\boldsymbol{3.204}^{\pm.027}$ | $10.80^{\pm.117}$ | $\boldsymbol{2.192}^{\pm.071}$

Table 2: We involve KIT [48] dataset and evaluate these state-of-the-art
methods on the text-to-motion task. Please refer to Tab. 1 for metrics
details. Figure 4: Qualitative comparison of unconditioned motion generation.
Samples for our MLD and ACTOR [46] trained on a split of AMASS [41] dataset,
the motion part of [17]. More samples in the supplements.

###  4.3 Comparisons on Text-to-motion

By introducing motion latent diffusion models based on text input $w^{1:N}$,
we open up the exploration of conditional motion generation. We train a 25M
parameter MLD-1 conditioned on the language prompt and employ the frozen CLIP
[52] model as $\tau_{\theta}^{w}$ to encode the text to projected pooled
output, giving $w_{clip}^{1}\in\mathbb{R}^{1,256}$. We evaluate state-of-the-
art methods on HumanML3D and KIT with suggested metrics [17] under the $95\%$
confidence interval from 20 times running. Most results are borrowed from
their own paper or the benchmark in [18], except TEMOS [47]. We train it with
the proposed default model setting on two datasets to uniform the evaluation
metrics. Besides, the deterministic methods [49, 5, 2] can not generate
diverse results from one input and thus we leave their MModality metrics
empty. Tab. 1 and Tab. 2 summarize the comparisons results. We achieve the
best FID, R Precision and MM Dist on HumanML3D and KIT, outperforming previous
cross-modal models as well as motion diffusion models. It indicates high-
quality motion and high text prompt matching, as also shown in Fig. 3. Our
generated results correctly match the text prompt while maintaining a rich
diversity of generated motions.

Methods | UESTC | HumanAct12
---|---|---
$\text{FID}_{\text{train}}\downarrow$ | $\text{FID}_{\text{test}}\downarrow$ | ACC$\uparrow$ | DIV$\rightarrow$ | MM$\rightarrow$ | $\text{FID}_{\text{train}}\downarrow$ | ACC $\uparrow$ | DIV$\rightarrow$ | MM$\rightarrow$
Real | $2.92^{\pm.26}$ | $2.79^{\pm.29}$ | $0.988^{\pm.001}$ | $33.34^{\pm.320}$ | $14.16^{\pm.06}$ | $0.020^{\pm.010}$ | $0.997^{\pm.001}$ | $6.850^{\pm.050}$ | $2.450^{\pm.040}$
ACTOR [46] | $20.5^{\pm 2.3}$ | $23.43^{\pm 2.20}$ | $0.911^{\pm.003}$ | $31.96^{\pm.33}$ | $14.52^{\pm.09}$ | $0.120^{\pm.000}$ | $0.955^{\pm.008}$ | $6.840^{\pm.030}$ | $2.530^{\pm.020}$
INR [7] | $\boldsymbol{9.55}^{\pm.06}$ | $15.00^{\pm.09}$ | $0.941^{\pm.001}$ | $31.59^{\pm.19}$ | $14.68^{\pm.07}$ | $0.088^{\pm.004}$ | $0.973^{\pm.001}$ | $6.881^{\pm.048}$ | $2.569^{\pm.040}$
MDM [68] | $9.98^{\pm 1.33}$ | $\boldsymbol{12.81}^{\pm 1.46}$ | $0.950^{\pm.000}$ | $33.02^{\pm.28}$ | $\boldsymbol{14.26}^{\pm.12}$ | $0.100^{\pm.000}$ | $\boldsymbol{0.990}^{\pm.000}$ | $6.680^{\pm.050}$ | $\boldsymbol{2.520}^{\pm.010}$
MLD (Ours) | $12.89^{\pm.109}$ | $15.79^{\pm.079}$ | $\boldsymbol{0.954}^{\pm.001}$ | $\boldsymbol{33.52}^{\pm.14}$ | $13.57^{\pm.06}$ | $\boldsymbol{0.077}^{\pm.004}$ | $0.964^{\pm.002}$ | $\boldsymbol{6.831}^{\pm.050}$ | $2.824^{\pm.038}$

Table 3: Comparison of action-conditional motion synthesis on UESTC [26] and
HumanAct12 [19] dataset: $\text{FID}_{\text{train}}$,
$\text{FID}_{\text{train}}$ indicate the evaluated splits. Accuracy (ACC) for
action recognition. Diversity (DIV), MModality (MM) for generated motion
diversity within each action label.

###  4.4 Comparisons on Action-to-motion

The action-conditioned task is given an input action label to generate
relevant motion sequences. We compare with ACTOR [46], INR [7] and MDM [68].
ACTOR and INR are transformer-based VAE models and focus on the action-
conditioned task, and MDM is a diffusion model using the same learnable action
embedding module as ours. We still provide 20 evaluations as introduced and
report FID scores on the training set and test set like [46] for comparison.
Tab. 3 shows the comparison on two datasets, UESTC [26] and HumanAct12 [19].
MLD achieves state-of-the-art accuracy and diversity on UESTC and competitive
results on HumanAct12, indicating that diffusion models in motion latent can
also benefit action-conditioned generation task.

Figure 5: Comparison of unconditional motion generation on part of AMASS [41]
dataset with the state-of-the-art methods. We provide both FID and Diversity
to evaluate generated motions.

###  4.5 Comparisons on Unconditional Generation

We then evaluate the generation effect of MLD by introducing unconditional
task on motions of HumanML3D [17], actually part of AMASS [41]. MLD supports
two manners for unconditional generation, latent sampling ($cf.$ Tab. 4) and
diffusion sampling. Here we focus on the evaluation of the latter and employ
FID and Diversity for motion quality and diversity. With the same process on
training and evaluations on the part of AMASS [41] data, we provide real
motion, ACTOR [46], and VPoser-t [43] and MDM [68] as our comparison
baselines, We employ the transformer VAE from ACTOR, then follow TEMOS [47] to
make it class-agnostic and set 6 heads/layers for transformers, $10^{-4}$ as
learning rate. To perform the temporal-based task, the input of VPoser-t is
modified as a motion of fixed length. MDM also supports this task, thus we
fine-tune and evaluate their provided model. Fig. 5 reports that MLD has the
best motion generation quality and diversity.

##  5 Ablation Studies

MLD comprises a motion VAE model $\mathcal{V}$ and latent diffusion models
$\epsilon_{\theta}$, and both influence its effect. We first focus on
$\mathcal{V}$ to evaluate its components with generation and reconstruction
metrics. Based on these $\mathcal{V}$, we evaluate MLDs in diffusion learning
aiming at text-to-motion and unconditional synthesis, and then report time
costs on inference.

Method | Reconstruction |  | Generation |
---|---|---|---|---
MPJPE $\downarrow$ | PAMPJPE$\downarrow$ | ACCL$\downarrow$ |  | FID$\downarrow$ | DIV$\rightarrow$ |
Real | - | - | - |  | $0.002{\color[rgb]{0.0600000000000001,0.46,1}}$ | $9.503{\color[rgb]{0.0600000000000001,0.46,1}}$ |
VPoser-t [43] | $75.6$ | $48.6$ | $9.3$ |  | $1.430{\color[rgb]{0.0600000000000001,0.46,1}}$ | $8.336{\color[rgb]{0.0600000000000001,0.46,1}}$ |
ACTOR [46] | $65.3$ | $41.0$ | $7.0$ |  | $0.341{\color[rgb]{0.0600000000000001,0.46,1}}$ | $9.569{\color[rgb]{0.0600000000000001,0.46,1}}$ |
Ours-7 ($\mathcal{V}$,skip,9 layers) | $\boldsymbol{14.7}$ | $\boldsymbol{8.9}$ | $\boldsymbol{5.1}$ |  | $\boldsymbol{0.017}{\color[rgb]{0.0600000000000001,0.46,1}}$ | $\boldsymbol{9.554}{\color[rgb]{0.0600000000000001,0.46,1}}$ |
Ours-1 ($z$,$\mathbb{R}^{1\times 256}$) | $54.4$ | $41.6$ | $8.3$ |  | $0.247{\color[rgb]{0.0600000000000001,0.46,1}}$ | $9.630{\color[rgb]{0.0600000000000001,0.46,1}}$ |
Ours-2 ($z$,$\mathbb{R}^{2\times 256}$) | $51.8$ | $37.8$ | $8.3$ |  | $0.166{\color[rgb]{0.0600000000000001,0.46,1}}$ | $9.626{\color[rgb]{0.0600000000000001,0.46,1}}$ |
Ours-5 ($z$,$\mathbb{R}^{5\times 256}$) | $24.3$ | $14.7$ | $5.8$ |  | $0.043{\color[rgb]{0.0600000000000001,0.46,1}}$ | $9.593{\color[rgb]{0.0600000000000001,0.46,1}}$ |
Ours-7 ($z$,$\mathbb{R}^{7\times 256}$) | $14.7$ | $8.9$ | $5.1$ |  | $0.017{\color[rgb]{0.0600000000000001,0.46,1}}$ | $9.554{\color[rgb]{0.0600000000000001,0.46,1}}$ |
Ours-10 ($z$,$\mathbb{R}^{10\times 256}$) | $17.3$ | $11.5$ | $5.8$ |  | $0.025{\color[rgb]{0.0600000000000001,0.46,1}}$ | $9.589{\color[rgb]{0.0600000000000001,0.46,1}}$ |
Ours-7 ($\mathcal{V}$,w/ skip) | $14.7$ | $8.9$ | $5.1$ |  | $0.017{\color[rgb]{0.0600000000000001,0.46,1}}$ | $9.554{\color[rgb]{0.0600000000000001,0.46,1}}$ |
Ours-7 ($\mathcal{V}$, w/o skip) | $18.5$ | $10.4$ | $5.6$ |  | $0.027{\color[rgb]{0.0600000000000001,0.46,1}}$ | $9.528{\color[rgb]{0.0600000000000001,0.46,1}}$ |
Ours-7 ($\mathcal{V}$, 7 layers) | $16.0$ | $10.2$ | $5.3$ |  | $0.022{\color[rgb]{0.0600000000000001,0.46,1}}$ | $9.593{\color[rgb]{0.0600000000000001,0.46,1}}$ |
Ours-7 ($\mathcal{V}$, 9 layers) | $14.7$ | $8.9$ | $5.1$ |  | $0.017{\color[rgb]{0.0600000000000001,0.46,1}}$ | $9.554{\color[rgb]{0.0600000000000001,0.46,1}}$ |
Ours-7 ($\mathcal{V}$, 11 layers) | $17.2$ | $11.2$ | $5.4$ |  | $0.021{\color[rgb]{0.0600000000000001,0.46,1}}$ | $9.533{\color[rgb]{0.0600000000000001,0.46,1}}$ |

Table 4: Evaluation of our VAE models $\mathcal{V}$ on the motion part of
HumanML3D [17] dataset: MPJPE and PAMPJPE are measured in millimeter. ACCL
indicates acceleration error. We evaluate FID and DIV the same as Tab. 1. From
top to down, we propose real reference, VPoser-t [43] and ACTOR [46] as
baselines, the evaluation on latent $z\in\mathbb{R}^{i\times 256}$, with (w/)
or without (w/o) skip connection, $\mathcal{V}$ with different number of
transformer layers.

Effectiveness of Latents in Motion Sequences Representation. We first ablate
several components of our VAE models $\mathcal{V}$ in a controlled setup,
studying the shape of latent $z$, skip connection, and the number of
transformer layers, as shown in Tab. 4. The most important variable of MLD,
the latent vector $z$, is a bridge between $\mathcal{V}$ and diffusion models
$\tau_{\theta}$. We lock the pose $x^{i}$ (one frame of motion) embedding
dimensionality to 256, which is the same as [47] , and explore
$z\in\mathbb{R}^{i\times 256}$, giving MLD-$i$. We then evaluate the skip
connection and transformer layers on the best MLD-7. All comparison baselines,
including ACTOR [46] and VPoser-t [43], follow the same training and
evaluation with our proposed MLD. Since the original VPoser can only handle
single frame pose, we modified it to a sequential manner with a fixed length.
The results in Tab. 4 demonstrate the effectiveness of our proposed VAEs over
others in the motion sequences representation.

Models | R Precision | FID$\downarrow$ | MM Dist.$\downarrow$ | Diversity$\rightarrow$ | MModality$\uparrow$
---|---|---|---|---|---
Top 3$\uparrow$
Real | $0.797^{\pm.002}$ | $0.002^{\pm.000}$ | $2.974^{\pm.008}$ | $9.503^{\pm.065}$ | -
MLD-1 ($z$, $\mathbb{R}^{1\times 256}$) | $0.772^{\pm.002}$ | $0.473^{\pm.013}$ | $3.196^{\pm.010}$ | $9.724^{\pm.082}$ | $2.413^{\pm.079}$
MLD-2 ($z$, $\mathbb{R}^{2\times 256}$) | $0.727^{\pm.003}$ | $0.585^{\pm.015}$ | $3.448^{\pm.011}$ | $9.084^{\pm.081}$ | $\boldsymbol{2.725}^{\pm.093}$
MLD-5 ($z$, $\mathbb{R}^{5\times 256}$) | $0.722^{\pm.003}$ | $1.554^{\pm.019}$ | $3.511^{\pm.008}$ | $8.424^{\pm.081}$ | $2.542^{\pm.080}$
MLD-7 ($z$, $\mathbb{R}^{7\times 256}$) | $0.731^{\pm.002}$ | $1.011{{}^{\pm.019}}$ | $3.415{{}^{\pm.008}}$ | $8.736{{}^{\pm.064}}$ | $2.463{{}^{\pm.089}}$
MLD-10 ($z$, $\mathbb{R}^{10\times 256}$) | $0.703^{\pm.003}$ | $1.716^{\pm.027}$ | $3.616^{\pm.012}$ | $8.606^{\pm.067}$ | $2.604^{\pm.087}$
MLD-1 ($\epsilon_{\theta}$, cross-att) | $0.592{{}^{\pm.004}}$ | $1.922{{}^{\pm.041}}$ | $4.480{{}^{\pm.015}}$ | $8.598{{}^{\pm.088}}$ | $3.768{{}^{\pm.126}}$
MLD-1 ($\epsilon_{\theta}$, concat) | $0.772{{}^{\pm.002}}$ | $0.473{{}^{\pm.013}}$ | $3.196{{}^{\pm.010}}$ | $9.724{{}^{\pm.082}}$ | $2.413{{}^{\pm.079}}$
MLD-1 ($\epsilon_{\theta}$, w/o skip) | $0.749{{}^{\pm.003}}$ | $0.784{{}^{\pm.015}}$ | $3.363{{}^{\pm.010}}$ | $9.568{{}^{\pm.093}}$ | $2.597{{}^{\pm.098}}$
MLD-1 ($\epsilon_{\theta}$, w/ skip) | $0.772{{}^{\pm.002}}$ | $0.473{{}^{\pm.013}}$ | $3.196{{}^{\pm.010}}$ | $9.724{{}^{\pm.082}}$ | $2.413{{}^{\pm.079}}$
MLD-1 ($\epsilon_{\theta}$, 5 layers) | $0.760{{}^{\pm.002}}$ | $\boldsymbol{0.314}{{}^{\pm.010}}$ | $3.259{{}^{\pm.009}}$ | $9.706{{}^{\pm.072}}$ | $2.635{{}^{\pm.085}}$
MLD-1 ($\epsilon_{\theta}$, 7 layers) | $0.771{{}^{\pm.003}}$ | $0.349{{}^{\pm.012}}$ | $3.199{{}^{\pm.012}}$ | $\boldsymbol{9.624}{{}^{\pm.062}}$ | $2.504{{}^{\pm.088}}$
MLD-1 ($\epsilon_{\theta}$, 9 layers) | $\boldsymbol{0.772}{{}^{\pm.002}}$ | $0.473{{}^{\pm.013}}$ | $\boldsymbol{3.196}{{}^{\pm.010}}$ | $9.724{{}^{\pm.082}}$ | $2.413{{}^{\pm.079}}$
MLD-1 ($\epsilon_{\theta}$, 11 layers) | $0.771{{}^{\pm.003}}$ | $0.402{{}^{\pm.011}}$ | $3.203{{}^{\pm.013}}$ | $9.876{{}^{\pm.088}}$ | $2.478{{}^{\pm.076}}$

Table 5: Evaluation of text-based motion synthesis on HumanML3D [17]: we use
metrics in Tab. 1 and provides real reference, the evaluation on latent $z$
($cf.$ $\mathcal{V}$ in Tab. 4), cross-attention or concatenation with
conditions $\tau_{\theta}$, with (w/) or without (w/o) skip connection,
$\epsilon_{\theta}$ with different number of transformer layers.

Methods | FID$\downarrow$ | Diversity$\rightarrow$ | Methods | FID$\downarrow$ | Diversity$\rightarrow$
---|---|---|---|---|---
Real | - | 9.503 | Real | - | 9.503
VPoser-t [43] | $36.65$ | $3.259$ | MLD-1 ($z$,$\mathbb{R}^{1\times 256}$) | $\boldsymbol{1.055}$ | $\boldsymbol{8.577}$
ACTOR [46] | $14.14$ | $5.123$ | MLD-2 ($z$,$\mathbb{R}^{1\times 256}$) | $4.408$ | $7.420$
MDM [68] | 8.848 | 6.429 | MLD-5 ($z$,$\mathbb{R}^{5\times 256}$) | $7.829$ | $6.247$
MLD-1 ($\epsilon_{\theta}$, w/o skip) | $2.575$ | $7.566$ | MLD-7 ($z$,$\mathbb{R}^{7\times 256}$) | $7.614$ | $6.233$
MLD-1 ($\epsilon_{\theta}$, w/ skip) | $\boldsymbol{1.055}$ | $\boldsymbol{8.577}$ | MLD-10 ($z$,$\mathbb{R}^{10\times 256}$) | $9.624$ | $6.194$

Table 6: Evaluation of unconditional motion generation. From left to right, we
evaluate the denoiser $\epsilon_{\theta}$ with (w/) or without (w/o) skip
connection and the latent $z$ ($cf.\mathcal{V}$ in Tab. 4)

Effectiveness of Latents in Motion Latent-based Diffusion Models. In Tab. 5,
we task the text-to-motion task as our focus and evaluate latent diffusion
models $\epsilon_{\theta}$, using the similar metrics in Tab. 1. MLd-$i$
denotes the shape of latent $z\in\mathbb{R}^{i\times 256}$. Importantly,
MLD-1, using the smallest latent, wins the best performance in most metrics.
After that, the evaluation on the components of $\epsilon_{\theta}$ is
provided, cross-att and concate represent the cross-attention or concatenation
for condition embedding $\epsilon_{\theta}(c)$. Interestingly, MDM [68] also
reports the encoder design by concatenating embedding is better. We find that
skip connection, which is important for images [4, 56], also provided
significant improvement in motion latent diffusion models, but MLDs using
different numbers of layers in $\epsilon_{\theta}$ achieve similar effects on
this dataset. We then evaluate the generation of MLD by diffusion sampling,
different from the generation in $\mathcal{V}$ by latent sampling ($cf.$ Tab.
4). As shown in Tab. 6, the MLD using the smallest latent and skip connection
outperforms others. The evaluation of how different language models influence
MLDs and the details of latent/diffusion sampling are provided in supplements.

Figure 6: Comparison of the inference time costs on text-to-motion. We
calculate AITS on the test set of HumanML3D [17] without model or data loading
parts. All tests are performed on the same Tesla V100. The closer the model is
to the origin the better.

Inference time. While diffusion models lead to significant improvements, one
notable limitation of motion diffusion models [68, 74] is the long inference
time. Although we have adopted Denoising diffusion implicit models (DDIM) [64]
to accelerate sampling, MDM [68] still requires 24.74 seconds for average on a
single Tesla V100. They also claim their inference time could be about a
minute. Compared to them, our MLD needs less computational overhead, with two
orders of magnitude faster while achieving higher performance.

##  6 Disscusion

As the trial to explore conditional motion generation with motion latent
diffusion models, the proposed MLD still owns limitations as follows. First,
same as most motion generation methods, our method can generate arbitrary
length results but still under the max-length in the dataset. It’s interesting
to model a non-stop human motion in temporal consistency. Besides, MLD focuses
on articulated human bodies, while there is also other work on faces [27, 6],
hands [58, 37, 36] and even animal [60, 79] motion.

We propose a motion latent-based diffusion model to generate plausible human
motion sequences conforming to the action classes or natural language
descriptions. Compared to the compatible cross-modal latent space-based
method, our MLD could produce more diverse and plausible human motion
sequences; Compared to the previous diffusion-based methods on raw motion
sequences, our MLD needs less computational overhead, with two orders of
magnitude faster. Extensive experiments on various human motion generation
tasks demonstrate the effectiveness and efficiency of our proposed MLD.

## References

  * [1] Hyemin Ahn, Timothy Ha, Yunho Choi, Hwiyeon Yoo, and Songhwai Oh.  Text2action: Generative adversarial synthesis from language to action.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5915–5920. IEEE, 2018.
  * [2] Chaitanya Ahuja and Louis-Philippe Morency.  Language2pose: Natural language grounded pose forecasting.  In 2019 International Conference on 3D Vision (3DV), pages 719–728. IEEE, 2019.
  * [3] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein generative adversarial networks.  In International conference on machine learning, pages 214–223. PMLR, 2017.
  * [4] Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu.  All are worth words: a vit backbone for score-based diffusion models.  arXiv preprint arXiv:2209.12152, 2022.
  * [5] Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha.  Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents.  In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pages 1–10. IEEE, 2021.
  * [6] Xuan Cao, Zhang Chen, Anpei Chen, Xin Chen, Shiying Li, and Jingyi Yu.  Sparse photometric 3d face reconstruction guided by morphable models.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4635–4644, 2018.
  * [7] Pablo Cervantes, Yusuke Sekikawa, Ikuro Sato, and Koichi Shinoda.  Implicit neural representations for variable length human motion generation.  In European Conference on Computer Vision, pages 356–372. Springer, 2022.
  * [8] Xin Chen, Anqi Pang, Wei Yang, Yuexin Ma, Lan Xu, and Jingyi Yu.  Sportscap: Monocular 3d human motion capture and fine-grained understanding in challenging sports videos.  International Journal of Computer Vision, 129(10):2846–2864, 2021\.
  * [9] Xin Chen, Zhuo Su, Lingbo Yang, Pei Cheng, Lan Xu, Bin Fu, and Gang Yu.  Learning variational motion prior for video-based motion capture.  arXiv preprint arXiv:2210.15134, 2022.
  * [10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [11] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  * [12] Yinglin Duan, Tianyang Shi, Zhengxia Zou, Yenan Lin, Zhehui Qian, Bohan Zhang, and Yi Yuan.  Single-shot motion completion with transformer.  arXiv preprint arXiv:2103.00776, 2021.
  * [13] Anindita Ghosh, Noshaba Cheema, Cennet Oguz, Christian Theobalt, and Philipp Slusallek.  Synthesis of compositional animations from textual descriptions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1396–1406, 2021.
  * [14] John C Gower.  Generalized procrustes analysis.  Psychometrika, 40(1):33–51, 1975.
  * [15] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville.  Improved training of wasserstein gans.  Advances in neural information processing systems, 30, 2017.
  * [16] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng.  Generating diverse and natural 3d human motions from text.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152–5161, June 2022.
  * [17] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng.  Generating diverse and natural 3d human motions from text.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152–5161, June 2022.
  * [18] Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng.  Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts.  In ECCV, 2022.
  * [19] Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng.  Action2motion: Conditioned generation of 3d human motions.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 2021–2029, 2020.
  * [20] Félix G Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal.  Robust motion in-betweening.  ACM Transactions on Graphics (TOG), 39(4):60–1, 2020.
  * [21] Yannan He, Anqi Pang, Xin Chen, Han Liang, Minye Wu, Yuexin Ma, and Lan Xu.  Challencap: Monocular 3d capture of challenging human performances using multi-modal references.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11400–11411, 2021.
  * [22] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  arXiv preprint arXiv:2210.02303, 2022.
  * [23] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [24] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * [25] Daniel Holden, Jun Saito, and Taku Komura.  A deep learning framework for character motion synthesis and editing.  ACM Transactions on Graphics (TOG), 35(4):1–11, 2016.
  * [26] Yanli Ji, Feixiang Xu, Yang Yang, Fumin Shen, Heng Tao Shen, and Wei-Shi Zheng.  A large-scale rgb-d database for arbitrary-view human action recognition.  In Proceedings of the 26th ACM international Conference on Multimedia, pages 1510–1518, 2018.
  * [27] Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen.  Audio-driven facial animation by joint end-to-end learning of pose and emotion.  ACM Transactions on Graphics (TOG), 36(4):1–12, 2017.
  * [28] Jihoon Kim, Jiseob Kim, and Sungjoon Choi.  Flame: Free-form language-based motion synthesis & editing.  arXiv preprint arXiv:2209.00349, 2022.
  * [29] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models.  Advances in neural information processing systems, 34:21696–21707, 2021.
  * [30] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [31] Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black.  Vibe: Video inference for human body pose and shape estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [32] Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz.  Dancing to music.  Advances in neural information processing systems, 32, 2019.
  * [33] Buyu Li, Yongchi Zhao, Shi Zhelun, and Lu Sheng.  Danceformer: Music conditioned 3d dance generation with parametric motion transformer.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1272–1279, 2022.
  * [34] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa.  Ai choreographer: Music conditioned 3d dance generation with aist++.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13401–13412, 2021.
  * [35] Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B Hashimoto.  Diffusion-lm improves controllable text generation.  2022\.
  * [36] Yuwei Li, Minye Wu, Yuyao Zhang, Lan Xu, and Jingyi Yu.  Piano: A parametric hand bone model from magnetic resonance imaging.  arXiv preprint arXiv:2106.10893, 2021.
  * [37] Yuwei Li, Longwen Zhang, Zesong Qiu, Yingwenqi Jiang, Nianyi Li, Yuexin Ma, Yuyao Zhang, Lan Xu, and Jingyi Yu.  Nimble: a non-rigid hand model with bones and muscles.  ACM Transactions on Graphics (TOG), 41(4):1–16, 2022.
  * [38] Xiao Lin and Mohamed R Amer.  Human motion modeling using dvgans.  arXiv preprint arXiv:1804.10652, 2018.
  * [39] Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel van de Panne.  Character controllers using motion vaes.  ACM Trans. Graph., 39(4), 2020.
  * [40] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  Smpl: A skinned multi-person linear model.  ACM Trans. Graph., 34(6):248:1–248:16, Oct. 2015.
  * [41] Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black.  Amass: Archive of motion capture as surface shapes.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [42] Christian Mandery, Ömer Terlemez, Martin Do, Nikolaus Vahrenkamp, and Tamim Asfour.  The kit whole-body human motion database.  In 2015 International Conference on Advanced Robotics (ICAR), pages 329–336. IEEE, 2015.
  * [43] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3d hands, face, and body from a single image.  In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [44] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3d hands, face, and body from a single image.  In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10975–10985, June 2019.
  * [45] Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa.  Amp: Adversarial motion priors for stylized physics-based character control.  ACM Trans. Graph., 40(4), July 2021.
  * [46] Mathis Petrovich, Michael J. Black, and Gül Varol.  Action-conditioned 3D human motion synthesis with transformer VAE.  In International Conference on Computer Vision (ICCV), 2021.
  * [47] Mathis Petrovich, Michael J. Black, and Gül Varol.  TEMOS: Generating diverse human motions from textual descriptions.  In European Conference on Computer Vision (ECCV), 2022.
  * [48] Matthias Plappert, Christian Mandery, and Tamim Asfour.  The kit motion-language dataset.  Big Data, 4(4):236–252, dec 2016.
  * [49] Matthias Plappert, Christian Mandery, and Tamim Asfour.  Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks.  Robotics and Autonomous Systems, 109:13–26, 2018.
  * [50] Abhinanda R. Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J. Black.  BABEL: Bodies, action and behavior with english labels.  In Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 722–731, June 2021.
  * [51] Sigal Raab, Inbal Leibovitch, Peizhuo Li, Kfir Aberman, Olga Sorkine-Hornung, and Daniel Cohen-Or.  Modi: Unconditional motion synthesis from diverse data.  arXiv preprint arXiv:2206.08010, 2022.
  * [52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [53] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [54] Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf.  Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting.  In International Conference on Machine Learning, pages 8857–8868. PMLR, 2021.
  * [55] Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J. Guibas.  Humor: 3d human motion model for robust pose estimation.  In International Conference on Computer Vision (ICCV), 2021.
  * [56] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and BjÃ¶rn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [57] Javier Romero, Dimitrios Tzionas, and Michael J. Black.  Embodied hands: Modeling and capturing hands and bodies together.  ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6), Nov. 2017.
  * [58] Javier Romero, Dimitrios Tzionas, and Michael J Black.  Embodied hands: Modeling and capturing hands and bodies together.  arXiv preprint arXiv:2201.02610, 2022.
  * [59] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [60] Nadine Rueegg, Silvia Zuffi, Konrad Schindler, and Michael J. Black.  BARC: Learning to regress 3D dog shape from images by exploiting breed information.  In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 3876–3884, June 2022.
  * [61] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [62] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [63] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  * [64] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * [65] Sebastian Starke, Ian Mason, and Taku Komura.  Deepphase: periodic autoencoders for learning motion phase manifolds.  ACM Transactions on Graphics (TOG), 41(4):1–13, 2022.
  * [66] Sebastian Starke, He Zhang, Taku Komura, and Jun Saito.  Neural state machine for character-scene interactions.  ACM Trans. Graph., 38(6):209–1, 2019.
  * [67] Ömer Terlemez, Stefan Ulbrich, Christian Mandery, Martin Do, Nikolaus Vahrenkamp, and Tamim Asfour.  Master motor map (mmm)—framework and toolkit for capturing, representing, and reproducing human motion on humanoid robots.  In 2014 IEEE-RAS International Conference on Humanoid Robots, pages 894–901. IEEE, 2014.
  * [68] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Amit H Bermano, and Daniel Cohen-Or.  Human motion diffusion model.  arXiv preprint arXiv:2209.14916, 2022.
  * [69] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [70] Timo von Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll.  Recovering accurate 3d human pose in the wild using imus and a moving camera.  In European Conference on Computer Vision (ECCV), sep 2018.
  * [71] Ziniu Wan, Zhengjia Li, Maoqing Tian, Jianbo Liu, Shuai Yi, and Hongsheng Li.  Encoder-decoder with multi-level attention for 3d human shape and pose estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13033–13042, 2021.
  * [72] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang.  Geodiff: A geometric diffusion model for molecular conformation generation.  In International Conference on Learning Representations, 2022.
  * [73] Sijie Yan, Zhizhong Li, Yuanjun Xiong, Huahan Yan, and Dahua Lin.  Convolutional sequence generation for skeleton-based action synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4394–4402, 2019.
  * [74] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu.  Motiondiffuse: Text-driven human motion generation with diffusion model.  arXiv preprint arXiv:2208.15001, 2022.
  * [75] Yan Zhang, Michael J Black, and Siyu Tang.  Perpetual motion: Generating unbounded human motion.  arXiv preprint arXiv:2007.13886, 2020.
  * [76] Yan Zhang, Michael J Black, and Siyu Tang.  We are more than our joints: Predicting how 3d bodies move.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3372–3382, 2021.
  * [77] Rui Zhao, Hui Su, and Qiang Ji.  Bayesian adversarial human motion synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6225–6234, 2020.
  * [78] Linqi Zhou, Yilun Du, and Jiajun Wu.  3d shape generation and completion through point-voxel diffusion.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5826–5835, October 2021.
  * [79] Silvia Zuffi, Angjoo Kanazawa, and Michael J. Black.  Lions and tigers and bears: Capturing non-rigid, 3D, articulated shape from images.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3955–3963. IEEE Computer Society, 2018.

##  Appendix

This appendix provides more qualitative results (Fig. 7), several additional
experiments (Sec. B) on the components of motion latent diffusion (MLD)
models, evaluations on hyperparameters (Sec. C), user study (Sec. D), details
of motion representations (Sec. E), implementation details of MLD models (Sec.
F) and metric definitions (Sec. G).

Video. We have provided supplemental videos in Project Page. In these
supplemental videos, we show 1) comparisons of text-based motion generation,
2) comparisons of action-conditional motion generation, and 3) more samples of
unconditional generation. We suggest the reader watch this video for dynamic
motion results.

Code is available on GitHub Page. We provide the process of the training and
evaluation of MLD models, the pre-trained model files, the demo script, and
example results.

##  A Qualitative Results

Figure 7: The comparison of the state-of-the-art methods on action-conditional
motion synthesis task. All provided methods are under the same training and
inference setting on HumanAct12 dataset [19]. We generate three motions for
each action label. The results demonstrate that our generations correspond
better to the action label and have richer diversity. Figure 8: More samples
from our best model for text-to-motion synthesis, MLD-1, which was trained on
the HumanML3D dataset. Samples generated with text prompts of the test set. We
recommend the supplemental video to see these motion results.

##  B Additional Experiments

We conduct several experiments to continue the evaluations of MLD models. We
first study the influence of language models $\tau_{\theta}^{w}$ and the shape
of text embedding on motion generations. Then, we evaluate the effectiveness
of long skip connections for motion diffusion models. We finally study the
importance of regularization on motion latent space.

###  B.1 Evaluation of Language Models $\tau_{\theta}^{w}$

We experiment with different language models, CLIP [52] and BERT [10].
Inspired by Stable Diffusion [56], we leverage the hidden state of CLIP to
generate word-wise tokens and explore its effects. The comparisons are listed
in Tab. 7. CLIP is more suited to our task compared to BERT, and the word-wise
text tokens are competitive with the single token, however, lower the
computation efficiency of diffusion models. Therefore, we choose CLIP and a
single text token for our models.

Models | Text Encoder | Embeddings | R Precision | FID$\downarrow$ | MM Dist$\downarrow$ | Diversity$\rightarrow$ | MModality$\uparrow$
---|---|---|---|---|---|---|---
$\tau_{\theta}^{w}$ | Shape | Top 3$\uparrow$
Real | - | - | $0.797^{\pm.002}$ | $0.002^{\pm.000}$ | $2.974^{\pm.008}$ | $9.503^{\pm.065}$ | -
MLD-1 | BERT [10] | $1\times 256$ | $0.725^{\pm.002}$ | $0.553^{\pm.020}$ | $3.530^{\pm.011}$ | $\boldsymbol{9.697}^{\pm.080}$ | $\boldsymbol{3.360}^{\pm.118}$
MLD-1 | CLIP [52] | $1\times 256$ | $\boldsymbol{0.769}^{\pm.002}$ | $0.446^{\pm.011}$ | $\boldsymbol{3.227}^{\pm.010}$ | $9.772^{\pm.071}$ | $2.413^{\pm.072}$
MLD-1 | CLIP [52] | $77\times 256$ | $0.737^{\pm.002}$ | $\boldsymbol{0.422}^{\pm.012}$ | $3.436^{\pm.010}$ | $9.840^{\pm.082}$ | $2.799^{\pm.107}$

Table 7: Quantitative comparison of the employed language models. Here we set
batch size to 500 and only change the text encoder $\tau_{\theta}^{w}$.

###  B.2 Effectiveness of Long Skip Connection

We have demonstrated the effectiveness of skip connection, especially on
diffusion models in Tab. 5. Here we analyze its influence on the training of
diffusion stage. As shown in Fig. 9, the model with long skip connection not
only achieves higher performance but also provides faster convergence compared
to the other one. The results suggest leveraging long skip connections for
iterative motion diffusion models.

Figure 9: The evaluation on long skip connection on diffusion training stage.
Two sub-figures are under the same training process and evaluated on the test
set of HumanML3D. Training steps indicate the epoch number.

###  B.3 Diffusion on Autoencoder or VAE

We study the importance of regularization on motion latent space. The
regularized latent space provides stronger generation ability and supports the
latent diffusion models as demonstrated:

Method | Reconstruction | Generation
---|---|---
MPJPE $\downarrow$ | PAMPJPE$\downarrow$ | ACCL$\downarrow$ | FID$\downarrow$ | DIV$\rightarrow$
Autoencoder | 38.5 | 28.2 | 5.8 | 0.156 | 9.628
VAE | $\boldsymbol{14.7}$ | $\boldsymbol{8.9}$ | $\boldsymbol{5.1}$ | $\boldsymbol{0.017}{\color[rgb]{0.0600000000000001,0.46,1}}$ | $\boldsymbol{9.554}{\color[rgb]{0.0600000000000001,0.46,1}}$
Method | R Precision | FID$\downarrow$ | MM Dist$\downarrow$ | Diversity$\rightarrow$ | MModality$\uparrow$
Top 3$\uparrow$
MLD w/ Autoencoder | $0.581^{\pm.003}$ | $5.033^{\pm.061}$ | $4.600^{\pm.018}$ | $7.953^{\pm.083}$ | $\boldsymbol{3.754}^{\pm.111}$
MLD w/ VAE | $\boldsymbol{0.772}^{\pm.002}$ | $\boldsymbol{0.473}^{\pm.013}$ | $\boldsymbol{3.196}^{\pm.010}$ | $\boldsymbol{9.724}^{\pm.082}$ | $2.413^{\pm.079}$

Table 8: Evaluation of autoencoder (without Kullback-Leibler regularization)
and VAE model on motion generations.

##  C Evaluation of Hyperparameters

Here, we present two different experiments of text-to-motion on HumanML3D
[17]. The first experiment is to change the dropout $p$ and scale in
classifier-free diffusion guidance [24]. In Tab. 9, we find that by changing
dropout $p$ from $0.1$ to $0.25$, the text correspondences (R Precision)
become worse but the motion quality (FID) gets better. It is the same as
changing scale $s$ range from $7.5$ to $2.5$. Besides, some settings like
$(0.25,7.5)$ achieve the best FID of 0.229, but we still suggest $(0.1,7.5)$
as dropout and scale $(p,s)$ for MLD models (Sec. 4) overall metrics.

Next, in Tab. 10, we experiment with batch sizes of 32, 64, 128, 256 and 512
under 8 Tesla V100 each with 32 GPU memory. We set it to 64 in our other
experiments.

Models | Classifier-free | R Precision | FID$\downarrow$ | MM Dist$\downarrow$ | Diversity$\rightarrow$ | MModality$\uparrow$
---|---|---|---|---|---|---
Dropout | Scale | Top 3$\uparrow$
Real | - | - | $0.797^{\pm.002}$ | $0.002^{\pm.000}$ | $2.974^{\pm.008}$ | $9.503^{\pm.065}$ | -
MLD-1 | $p=0.05$ | $s=7.5$ | $0.766^{\pm.002}$ | $0.574^{\pm.013}$ | $3.237^{\pm.007}$ | $9.664^{\pm.069}$ | $2.433^{\pm.074}$
MLD-1 | $p=0.10$ | $s=7.5$ | $\boldsymbol{0.772}^{\pm.002}$ | $0.473^{\pm.013}$ | $\boldsymbol{3.196}^{\pm.010}$ | $9.724^{\pm.082}$ | $2.413^{\pm.079}$
MLD-1 | $p=0.15$ | $s=7.5$ | $0.765^{\pm.002}$ | $0.311^{\pm.009}$ | $3.209^{\pm.007}$ | $9.649^{\pm.065}$ | $2.525^{\pm.070}$
MLD-1 | $p=0.20$ | $s=7.5$ | $0.761^{\pm.002}$ | $0.279^{\pm.011}$ | $3.243^{\pm.009}$ | $\boldsymbol{9.632}^{\pm.082}$ | $2.651^{\pm.080}$
MLD-1 | $p=0.25$ | $s=7.5$ | $0.757^{\pm.002}$ | $\boldsymbol{0.229}^{\pm.010}$ | $3.260^{\pm.008}$ | $9.649^{\pm.069}$ | $\boldsymbol{2.685}^{\pm.084}$
MLD-1 | $p=0.30$ | $s=7.5$ | $0.759^{\pm.002}$ | $0.289^{\pm.010}$ | $3.249^{\pm.008}$ | $9.670^{\pm.073}$ | $2.650^{\pm.082}$
MLD-1 | $p=0.10$ | $s=1.5$ | $0.648^{\pm.002}$ | $0.401^{\pm.019}$ | $3.857^{\pm.009}$ | $9.263^{\pm.056}$ | $\boldsymbol{3.914}^{\pm.115}$
MLD-1 | $p=0.10$ | $s=2.5$ | $0.720^{\pm.002}$ | $\boldsymbol{0.350}^{\pm.013}$ | $3.441^{\pm.010}$ | $\boldsymbol{9.549}^{\pm.058}$ | $3.201^{\pm.098}$
MLD-1 | $p=0.10$ | $s=3.5$ | $0.745^{\pm.002}$ | $0.358^{\pm.011}$ | $3.299^{\pm.009}$ | $9.639^{\pm.065}$ | $2.890^{\pm.087}$
MLD-1 | $p=0.10$ | $s=4.5$ | $0.758^{\pm.002}$ | $0.375^{\pm.011}$ | $3.232^{\pm.009}$ | $9.676^{\pm.069}$ | $2.701^{\pm.078}$
MLD-1 | $p=0.10$ | $s=5.5$ | $0.764^{\pm.002}$ | $0.396^{\pm.011}$ | $3.202^{\pm.010}$ | $9.681^{\pm.072}$ | $2.577^{\pm.076}$
MLD-1 | $p=0.10$ | $s=6.5$ | $0.767^{\pm.002}$ | $0.424^{\pm.011}$ | $\boldsymbol{3.191}^{\pm.009}$ | $9.658^{\pm.072}$ | $2.498^{\pm.074}$
MLD-1 | $p=0.10$ | $s=7.5$ | $\boldsymbol{0.772}^{\pm.002}$ | $0.473^{\pm.013}$ | $3.196^{\pm.010}$ | $9.724^{\pm.082}$ | $2.413^{\pm.079}$
MLD-1 | $p=0.10$ | $s=8.5$ | $0.768^{\pm.002}$ | $0.504^{\pm.012}$ | $3.207^{\pm.009}$ | $9.604^{\pm.073}$ | $2.413^{\pm.072}$
MLD-1 | $p=0.10$ | $s=9.5$ | $0.766^{\pm.001}$ | $0.555^{\pm.012}$ | $3.227^{\pm.010}$ | $9.567^{\pm.072}$ | $2.394^{\pm.069}$

Table 9: Classifier-free Diffusion Guidance: We study the influence of its
hyperparameters, dropout $p$ and scale $s$ on text-to-motion.

Models | Batch Size | R Precision | FID$\downarrow$ | MM Dist$\downarrow$ | Diversity$\rightarrow$ | MModality$\uparrow$
---|---|---|---|---|---|---
Top 3$\uparrow$
Real | - | $0.797^{\pm.002}$ | $0.002^{\pm.000}$ | $2.974^{\pm.008}$ | $9.503^{\pm.065}$ | -
MLD-1 | 32 | $0.761^{\pm.003}$ | $0.445^{\pm.012}$ | $3.243^{\pm.010}$ | $9.751^{\pm.086}$ | $\boldsymbol{2.581}^{\pm.070}$
MLD-1 | 64 | $\boldsymbol{0.772}^{\pm.002}$ | $0.473^{\pm.013}$ | $\boldsymbol{3.196}^{\pm.010}$ | $9.724^{\pm.082}$ | $2.413^{\pm.079}$
MLD-1 | 128 | $0.771^{\pm.002}$ | $\boldsymbol{0.421}^{\pm.013}$ | $3.187^{\pm.008}$ | $\boldsymbol{9.691}^{\pm.080}$ | $2.370^{\pm.078}$
MLD-1 | 256 | $0.770^{\pm.002}$ | $0.423^{\pm.010}$ | $3.211^{\pm.007}$ | $9.800^{\pm.070}$ | $2.401^{\pm.074}$
MLD-1 | 512 | $0.769^{\pm.002}$ | $0.446^{\pm.011}$ | $3.227^{\pm.010}$ | $9.772^{\pm.071}$ | $2.413^{\pm.072}$

Table 10: Batch Size: We explore the evaluation of the batch size. We find the
results are close and suggest 64 and 128 in this task. Figure 10: User Study:
Each bar indicates the preference rate of MLD over other methods. The red line
indicates the 50%. Please refer to the supplemental video for the comparisons
of dynamic motion results.

##  D User Study

For the pairwise comparisons of the user study presented in Fig. 10, we use
the force-choice paradigm to ask “Which of the two motions is more realistic?”
and “which of the two motions corresponds better to the text prompt?”. The
provided motions are generated from 30 text descriptions, which are randomly
generated from the test set of HumanML3D[17] dataset. We invite 20 users and
provide three comparisons, ours and MDM [68], ours and T2M [16], ours and real
motions from the dataset. Our MLD was preferred over the other state-of-the-
art methods and even competitive to the ground troth motions.

##  E Motion Representations

Four relevant motion representations are summarized:

HumanML3D Format [17] proposes a motion representation $x^{1:L}$ inspired by
motion features in character control [66, 45, 65]. This redundant
representation is quite suited to neural models, particularly variational
autoencoders. Specifically, the $i$-th pose $x^{i}$ is defined by a tuple of
root angular velocity $\dot{r}^{a}\in\mathbb{R}$ along Y-axis, root linear
velocities $(\dot{r}^{x},\dot{r}^{z}\in\mathbb{R})$ on XZ-plane, root height
$r^{y}\in\mathbb{R}$, local joints positions
$\mathbf{j}^{p}\in\mathbb{R}^{3N_{j}}$, velocities
$\mathbf{j}^{v}\in\mathbb{R}^{3N_{j}}$ and rotations
$\mathbf{j}^{r}\in\mathbb{R}^{6N_{j}}$ in root space, and binary foot-ground
contact features $\mathbf{c}^{f}\in\mathbb{R}^{4}$ by thresholding the heel
and toe joint velocities, where $N_{j}$ denotes the joint number, giving:

$\displaystyle
x^{i}=\\{\dot{r}^{a},\dot{r}^{x},\dot{r}^{z},r^{y},\mathbf{j}^{p},\mathbf{j}^{v},\mathbf{j}^{r},\mathbf{c}^{f}\\}.$
(5)

SMPL-based Format [40]. The most popular parametric human model, SMPL [40] and
its variants [57, 44] propose motion parameters $\theta$ and shape parameters
$\beta$. $\theta\in\mathbb{R}^{3\times 23+3}$ is rotation vectors for 23
joints and a root, and $\beta$ are the weights for linear blended shapes. This
representation is popular in markerless motion capture [21, 8, 31]. By
involving the global translation $r$, the representation is formulated as:

$\displaystyle x^{i}=\\{r,\theta,\beta\\}.$ (6)

MMM Format [67]. Master Motor Map (MMM) representations propose joints angle
parameters by adopting a uniform skeleton structure with 50 DoFs. And most
recent methods [2, 13, 47] on text-to-motion task followed preprocess
procedure in [25] which transform joint rotation angles to $J=21$ joints XYZ
coordinates, giving $p_{m}\in\mathbb{R}^{3J}$, and global trajectory
$t_{root}$ for the root joint. The preprocessed representation can be
formulated as

$\displaystyle x^{i}=\\{p_{m},t_{root}\\}.$ (7)

Latent Format [40]. Latent representations are widely used in neural models
[46, 47, 19, 9]. We recognize it as motion representation in latent space. By
leveraging VAE models, latent vectors can represent plausible motions as:

$\displaystyle\hat{x}^{1:L}=\mathcal{D}(z),z=\mathcal{E}(x^{1:L})$ (8)

##  F Details on Motion Latent Diffusion Models

###  F.1 Details Information on Variational Autoencoder Models

We take HumanML3D [17] and its motion representation (Sec. E) as an example
here to explain our loss details of Variational Autoencoder Models
$\mathcal{V}$. The motion $x^{1:L}$ includes joint features and is supervised
with data term by mean squared error:

$\displaystyle\mathcal{L}_{data}=\left\|x^{1:L}-\mathcal{D}(\mathcal{E}(x^{1:L}))\right\|^{2}.$
(9)

To regularize latent space as a standard variational autoencoder [30], we
employ a Kullback-Leibler term between
$q(z|x^{1:L})=\mathcal{N}\left(z;\mathcal{E}_{\mu},\mathcal{E}_{\sigma^{2}}\right)$
and a standard Gaussion distribution $\mathcal{N}\left(z;0,1\right)$. The full
training loss of the VAE model $\mathcal{V}$ follows:

$\displaystyle\mathcal{L}_{\mathcal{V}}=\mathcal{L}_{data}(x^{1:L},\mathcal{D}(\mathcal{E}(x^{1:L})))+\lambda_{reg}\mathcal{L}_{reg}(x^{1:L};\mathcal{E},\mathcal{D}),$
(10)

where $\lambda_{reg}$ is a low weight to control the regularization. The KIT
[48], HumanAct12 [19] and UESTC [26] dataset processed by [47, 46] also
supports SMPL-based [40] motion representation. Here we list the loss terms
for this representation. The data term formulates as followed:

$\displaystyle\mathcal{L}_{data}=\sum_{i=1}^{L}\left\|{r}^{i}-\hat{{r}}^{i}\right\|_{2}+\sum_{i=1}^{L}\left\|{\theta}^{i}-\hat{{\theta}}^{i}\right\|_{2}+\left\|{\beta}-\hat{{\beta}}\right\|_{2}.$
(11)

Here the motion is $x^{1:L}=\\{{r}^{i},{\theta}^{i},{\beta}\\}^{L}_{i=1}$,
which includes global translation ${r}^{i}$, pose parameter ${\theta}^{i}$ and
shape parameter ${\beta}$ of the $i$-th frame. To enhance the full-body
supervision, the reconstruction term on the SMPL vertices follows:

$\displaystyle\mathcal{L}_{\mathrm{mesh}}=\sum_{i=1}^{L}\left\|{V}_{i}-M(\hat{{r}}^{i},\hat{{\theta}}^{i},\hat{{\beta}}^{i})\right\|^{2},$
(12)

where the body reconstruction function $M(\cdot)$ is from the differentiable
SMPL layer, while the vertices ${V}_{i}$ are calculated with the ground truth
motion parameters using the same layer. The reconstruction loss builds global
supervision on almost all predicted parameters
$\\{{r}_{t},{\theta}_{t},{\beta}\\}$ and shows a reliable supervision [46] for
motion generation. The full objective on SMPL-based motion representation
reads:

$\displaystyle\mathcal{L}_{\mathcal{V}}=\mathcal{L}_{data}+\lambda_{mesh}\mathcal{L}_{mesh}+\lambda_{reg}\mathcal{L}_{reg}.$
(13)

where $\lambda_{mesh}$ is the weight to enhance the supervision on the full-
body vertices. Besides, the regularization term is the same as the Kullback-
Leibler term in Eq. 11. In practice, the shape parameters, as part of global
motion features, increase the complexity of motion generation and influence
joint positions. We finally utilize the objective of Eq. 11 to train our text-
based models and Eq. 13 to train action-based models in comparisons and
evaluations.

###  F.2 Network Architectures

The details of network architecture are shown as Fig. 11, our MLD comprises
three main components, motion encoder $\mathcal{E}$, motion decoder
$\mathcal{D}$ and latent denoiser $\epsilon_{\theta}$. Please refer to the
following figure and Tab. 11 for more details.

Figure 11: Network architecture of our conditional MLD. We explain each
component in its right bottom part and the loss terms in Sec. F.1.

###  F.3 Implementation Details

For the experiments on text-to-motion, action-to-motion and unconditional
motion synthesis, we implement MLDs with various latent shapes as follows.

| MLD-1 | MLD-2 | MLD-5 | MLD-7 | MLD-10
---|---|---|---|---|---
$z$-shape | $1\times 256$ | $2\times 256$ | $5\times 256$ | $7\times 256$ | $10\times 256$
Training Diffusion steps | 1000 | 1000 | 1000 | 1000 | 1000
Inference Diffusion steps | 50 | 50 | 50 | 50 | 50
Noise Schedule | scaled linear | scaled linear | scaled linear | scaled linear | scaled linear
Denoiser Heads Number | 4 | 4 | 4 | 4 | 4
Denoiser Transformer Layers | 9 | 9 | 9 | 9 | 9
Conditioning | concat | concat | concat | concat | concat
Embedding Dimension | 256 | 256 | 256 | 256 | 256
VAE Heads Number | 4 | 4 | 4 | 4 | 4
VAE Transformer Layers | 9 | 9 | 9 | 9 | 9
Model Size (w/o clip) | 26.9M | 26.9M | 26.9M | 26.9M | 26.9M
Diffusino Batch Size | 64 | 64 | 64 | 64 | 64
Diffusion Epochs | 2000 | 2200 | 2400 | 2600 | 2800
VAE Batch Size | 128 | 128 | 128 | 128 | 128
VAE Epochs | 4000 | 4500 | 5000 | 5500 | 6000
Learning Rate | 1e-4 | 1e-4 | 1e-4 | 1e-4 | 1e-4

Table 11: Hyperparameters for the conditional MLDs in experiments. We train
these models on 8 Tesla V100. The smaller latent shape lowers the
computational requirements and provides faster inference.

##  G Metric Definitions

We provide more details of evaluation metrics in Sec. 4.1 as follows.

Motion Quality. Frechet Inception Distance (FID) is our principal metric to
evaluate the distribution similarity between generated and real motions,
calculated with the suitable feature extractor [19, 46, 16] for each dataset.
Besides, to evaluate the motion reconstruction error of VAEs, we use popular
metrics in motion capture [31, 8, 70], MPJPE, and PAMPJPE [14] for global and
local errors in millimeter, Acceleration Error (ACCL) for the quality on
temporal.

Generation Diversity. Following [19, 18], we use Diversity (DIV) and
MultiModality (MM) to measure the motion variance across the whole set and the
generated motion diversity within each text input separately. Here we take the
text-to-motion task as an example to explain the calculation steps and for
other tasks the operations are similar. To evaluate Diversity, all generated
motions are randomly sampled to two subsets of the same size $X_{d}$ with
motion feature vectors $\\{x_{1},..,x_{X_{d}}\\}$ and
$\\{x^{{}^{\prime}}_{1},..,x^{{}^{\prime}}_{X_{d}}\\}$ respectively. Then
diversity is formalized as:

$\text{DIV}=\frac{1}{X_{d}}\sum_{i=1}^{X_{d}}||x_{i}-x^{{}^{\prime}}_{i}||.$

To evaluate MultiModality, a set of text descriptions with size $J_{m}$ is
randomly sampled from all descriptions. Then two subsets of the same size
$X_{m}$ are randomly sampled from all motions generated by $j\text{-th}$ text
descriptions, with motion feature vectors $\\{x_{j,1},..,x_{j,X_{m}}\\}$ and
$\\{x^{{}^{\prime}}_{j,1},..,x^{{}^{\prime}}_{j,X_{m}}\\}$ respectively. The
multimodality is calculated as:

$\text{MM}=\frac{1}{J_{m}\times
X_{m}}\sum_{j=1}^{J_{m}}\sum_{i=1}^{X_{m}}||x_{j,i}-x^{{}^{\prime}}_{j,i}||.$

Condition Matching. For the text-to-motion task, [16] provides motion/text
feature extractors to produce geometrically closed features for matched text-
motion pairs, and vice versa. Under this feature space, motion-retrieval
precision (R Precision) first mix generated motion with 31 mismatched motions
and then calculates the text-motion top-1/2/3 matching accuracy, and Multi-
modal Distance (MM Dist) that calculates the distance between generated
motions and text. For action-to-motion, for each dataset a pretrained
recognition model [19, 46] is used to calculate the average motion Accuracy
(ACC) for action categories.

Time Costs. To evaluate the computing efficiency of diffusion models,
especially the inference efficiency, we propose Average Inference Time per
Sentence (AITS) measured in seconds. In our case, we calculate AITS ($cf.$
Fig. 6) on the test set of HumanML3D [17], set the batch size to one, and
ignore the time cost for model and dataset loading parts.


Let $(\relstr A, \relstr B)$ be a Boolean \kl(PCSP){template}. If $\Pol(\relstr A, \relstr B)$ contains $\MAX$ or $\MIN$, then $\PCSP(\relstr A, \relstr B)$ (in the search version) is in P.
\end{theorem}

\begin{theorem}[\cite{brakensiek2017promise,ficak2019symmetric}]\label{thm:atEZ}
Let $(\relstr A, \relstr B)$ be a Boolean \kl(PCSP){template}. If $\AT \subseteq \Pol(\relstr A, \relstr B)$, then $\PCSP(\relstr A, \relstr B)$ is in P.
\end{theorem}

\begin{theorem}[\cite{brakensiek2017promise,ficak2019symmetric}]\label{thm:thrEZ}
Let $(\relstr A, \relstr B)$ be a Boolean \kl(PCSP){template}. If $\THR{t} \subseteq \Pol(\relstr A, \relstr B)$ for some $t$, then $\PCSP(\relstr A, \relstr B)$ is in P.
\end{theorem}

\noindent For the threshold \kl{minions} \cite{brakensiek2017promise,ficak2019symmetric} provide an algorithm only under assumption $t \in \mathbb{Q}$. The remaining part of the section is devoted to a small adjustment to the aforementioned algorithm in case $t \notin \mathbb{Q}$.

We follow the algorithm described in \cite{brakensiek2017promise} closely. It is based on a linear programming relaxation approach.

\begin{definition}
Given an instance $\relstr I$ of a Boolean $\PCSP(\relstr A, \relstr B)$,
the \emph{basic linear programming (BLP) relaxation} of $\relstr I$ is the following linear program:
the variables are $x_v$ for every $v \in \relstr I$, while the linear constraints are as follows
\begin{itemize}
\item For each variable $v \in \relstr I$, stipulate that $0 \le x_v \le 1$
\item For each constraint $(\tuple v, R) \in \relstr I$, stipulate that $(x_{v_1}, \dots, x_{v_k})$ is in the convex hull of the elements of $R^{\relstr A}$
\end{itemize}
Note that a rational solution to such BLP instance (or reporting its absence) can be found in polynomial time.
\end{definition}

\begin{proof}[Proof of \cref{thm:thrEZ} when $t \notin \mathbb{Q}$]
Let $\relstr I$ be an input \kl{structure} of $\PCSP(\relstr A, \relstr B)$ over the set of variables $[n]$.
In the search version, the goal is to find a \kl{homomorphism} $\relstr I \to \relstr B$, given that a \kl{homomorphism} $\relstr I \to \relstr A$ exists (though, it's not provided).

The algorithm goes as follows:
\begin{itemize}
\item Construct the BLP relaxation;
\item Let $1$ be any variable in $\relstr I$;
\item Fix $x_1 = 0$, and re-solve the BLP,
\item If no solutions, fix $x_1 = 1$, and re-solve the BLP.
\end{itemize}

Assuming that $\relstr I \to \relstr A$, this procedure provides a rational solution $w \in [0, 1]^n$ such that $w_1 \in \Boolean$.
We claim that the map
\begin{equation*}
h_t(i) = \funfor{1}{t}(w_i) = \begin{cases}
1, &\quad\text{if } w_i > t \\
0, &\quad\text{if } w_i < t
\end{cases}
\end{equation*}
is a \kl{homomorphism} $\relstr I \to \relstr B$.

Consider any constraint $((v_1, \dots, v_k), R)$ and let $r = |R^{\relstr A}|$. Enumerate the elements $y^1, \dots, y^r \in R^{\relstr A}$.
Since $w$ is a rational solution to the LP, there exist $\alpha_1, \dots, \alpha_r \in \mathbb{Q}_{[0, 1]}$
such that $\sum \alpha_i = 1$ and $(w_{v_1}, \dots, w_{v_k}) = \sum \alpha_i \cdot y^i$.
Let $N \in \mathbb{N}$ be the common denominator of $\alpha_1, \dots, \alpha_r$.
Since a threshold $t$ function $\genthr{t}{N}$ of arity $N$ is a polymorphism of $(\relstr A, \relstr B)$, we have that
\begin{equation*}
\tuple z := \genthr{t}{N}(
\underbrace{y^1, \dots, y^1}_{\alpha_1 N},
\underbrace{y^2, \dots, y^2}_{\alpha_2 N},
\dots,
\underbrace{y^r, \dots, y^r}_{\alpha_r N}) \in R^{\relstr B}
\end{equation*}
On the other hand, note that
\begin{align*}
w_{v_j} > t &\implies \sum_i \alpha_i \cdot y^i_j = \frac{\sum_i \alpha_i N \cdot y^i_j}{N} > t \implies z_{v_j} = 1 \\
w_{v_j} < t &\implies \sum_i \alpha_i \cdot y^i_j = \frac{\sum_i \alpha_i N \cdot y^i_j}{N} < t \implies z_{v_j} = 0
\end{align*}
for every $1 \le j \le k$. In other words, $\tuple z = (h_t(v_1), \dots, h_t(v_k)) \in R^{\relstr B}$.

Hence $h_t$ is indeed a \kl{homomorphism} $\relstr I \to \relstr B$.
However, it isn't obvious how to compute the value $h_t(i)$ in polynomial time, since $t$ may be an arbitrary non-rational number.
Observe that there exists $q \in \{0, w_1, \dots, w_n\}$ such that
\begin{equation*}
\forall{i} \+\+ w_i > t \iff w_i > q
\end{equation*}
Consequently, the functions $\funfor{1}{t}$ and $\funforleq{1}{q}$ are equal on the set $\{w_1, \dots, w_n\}$.
The idea is now to compute the maps $h_q$ for each $q \in \{0, w_1, \dots, w_n\}$ (at most $n+1$ of them).
At least one of them is a valid \kl{homomorphism} $\relstr I \to \relstr B$ by the discussion above.
Since we can verify whether $h_q$ is a valid \kl{homomorphism} in a polynomial time, and there is $O(n)$ maps $h_q$ to consider,
the running time of the algorithm remains polynomial.
\end{proof}

\end{document}