# Learning Joint Representation of Human Motion and Language

Jihoon Kim,1 * Youngjae Yu, 2 * Seungyoun Shin, 1 Taehyun Byun, 1 Sungjoon
Choi 1

###### Abstract

In this work, we present MoLang (a Motion-Language connecting model) for
learning joint representation of human motion and language, leveraging both
unpaired and paired datasets of motion and language modalities. To this end,
we propose a motion-language model with contrastive learning, empowering our
model to learn better generalizable representations of the human motion
domain. Empirical results show that our model learns strong representations of
human motion data through navigating language modality. Our proposed method is
able to perform both action recognition and motion retrieval tasks with a
single model where it outperforms state-of-the-art approaches on a number of
action recognition benchmarks.

##  1 Introduction

As humans, we can easily generalize and imitate human motion and even
understand the contexts of human motion in the wild. However, it will be a
long and detour way for an autonomous agent (e.g., a humanoid robot) to learn
how to understand motions from limited motion capture data. Nevertheless, a
humanoid capable of representing and expressing its detailed motion holds
practical promise. With the advent of scalable Transformer (Vaswani et al.
2017), significant progress has been made in joining multiple modalities such
as image-text (Zhang et al. 2020; Radford et al. 2021; Jia et al. 2021; Tan
and Bansal 2019; Li et al. 2019; Lu et al. 2019; Chen et al. 2020; Yu et al.
2020), video/audio-text (Rouditchenko et al. 2020; Wang et al. 2021; Akbari et
al. 2021; Xu et al. 2021; Zellers et al. 2021) over the last few years. It has
been notable that the use of pre-trained models is effective for a variety of
visual downstream tasks. However, this paradigm has been adopted to a lesser
extent in human motion.

We postulate that a pre-trained human motion model is not extensively used for
the following reasons. First, unlike text or image data, which can be
retrieved relatively readily on the web, estimating the human motion skeleton
requires a significant amount of effort. While developments in pose estimation
have allowed us to infer human motions from images, there still remains a
hurdle in that the 3D skeleton estimated from a 2D image is not accurate, and
the quality varies substantially depending on the data. On the other hand,
motion capture devices can obtain highly accurate motion information, but
there is a limitation in that such devices are expensive and require
professional efforts for collecting a sufficient amount of clean motions.
Another problem is the ambiguity of language description for motion. Human
motion can semantically be interpreted in various ways. For example, “touching
a right toe”, “bending the waist”, and “touching the floor” are all valid
descriptions for a single motion, let alone the polysemy in language.

Figure 1: MoLang (a Motion-Language transformer model) learns the transferable
representation of human motion and language via contrastive learning with its
emphasis on achieving generalizable motion representation guided by a free-
form language.

In this work, we study these questions: Can the limited human motion data in
our hands be generalized through free-formed language annotations? Can the
model be adaptable with human motion and language in the wild (e.g. YouTube
(Miech et al. 2019; Zellers et al. 2021), Movie film (Marszałek, Laptev, and
Schmid 2009; Sigurdsson et al. 2016; Sigurdsson, Russakovsky, and Gupta 2017),
DVS (Rohrbach et al. 2016) etc.) to improve human motion perception in
context? Locating and curating valid pairs of estimated human motion and free-
form languages with a foundation model is an essential step forward.

As our first step, we introduce a new model MoLang that learns self-supervised
human motion representations and further improves through navigating
linguistic modalities. Enhanced with contrastive learning objective (Sohn
2016; Van den Oord, Li, and Vinyals 2018) to connect motion and language,
MoLang collaboratively learns human motion knowledge across modalities and
benefits from free-formed human motion datasets spanning classification and
description without format restrictions.

Figure 2: MoLang pivots human motion and arbitrary text via solving alignment
problems on both motion frames and words. Accordingly, the motion-Language co-
occurrences relationships are abundantly available on the public datasets and
web to potentially support training, strengthening our approach’s potential
directions.

As shown in Figure 2, MoLang seamlessly train heterogeneous motion labels
(e.g. motion description, action class) by turning them into distilled
language tokens learned by language modeling objectives (Devlin et al. 2018).
Thereby, the representation can be transferred to any human motion perception
task. For robust motion representation, we adopted Masked Motion Prediction
and Graph Convolutional Bottleneck to effectively align hierarchy inside a
sequence of human motion structure and free-formed linguistic annotation.

The Masked Motion Prediction is a self-supervision objective for human motion
data. Considering the motion characteristics, our model learns by predicting
randomly masked sub-sequence of motion without supervision. Beyond local sub-
sequence prediction, we give additional hierarchy to motion encoding via Graph
Convolutional Bottleneck. We create an information bottleneck with adjacent
joints to give locality by interpreting the human skeleton as a graph, which
improves the sample efficiency of motion encoding. By combining these methods,
we show that our model can effectively align several thousands of language
phrases with motion, beyond distinguishing a few categorical labels of the
human motion domain.

Experimental results show that MoLang learns robust human motion
representation and significantly improves motion downstream tasks. Our model
sets state-of-the-art skeleton-based action recognition tasks; HumanAct12 (Guo
et al. 2020), NTU-RGBD-13 (Guo et al. 2020), and UESTC (Ji et al. 2019).

Finally, ablations study and analysis of MoLang in section 4 show that 1)
multimodal alignment on motion works better when the model is trained on both
motion reconstruction and contrastive cross-modal matching. 2) using a diverse
set of language labels improves downstream performance compared to pre-
training on the only categorical label for motion, and 3) MoLang estimates not
only motion similarity but also alignment score for motion-language pair,
which sets a reliable foundation for unseen human motion estimation/detection
in the wild. The combination of these experimental results suggests that
integrating self-supervised motion reconstruction and multimodal alignment is
a promising path forward for future research on human motion understanding.

Overall, the study suggests that our newly proposed Masked Motion Prediction
objective and Graph Convolutional Bottleneck enhances self-supervised
contextualized motion representation to be more robust. Moreover, a mixture of
motion-language contrastive objective with auxiliary motion reconstruction
loss, which we call CstAR, allows free-formed language annotations to guide
the motion representation even more generalizable to downstream tasks.

Our key contributions are summarized as follows:

  1. 1.

We present MoLang (a Motion-Language transformer), enabling unified training
of large-scale human motion datasets spanning classification and description
generation as a unified resource.

  2. 2.

A new auxiliary reconstruction objective (Contrastive Auxiliary Reconstruction
(CstAR) for contrastive learning on motion and free-formed language, allowing
our model to learn better generalizable representation.

  3. 3.

A new self-supervised learning objective Masked Motion Prediction (MMP) and an
embedding module Graph Convolution Bottleneck (GCB) for human motion modeling
enable our model to navigate human joint structures efficiently.

  4. 4.

Experiments, ablations, and qualitative analysis demonstrate our model
outperforms state-of-the-art results. We also present a novel motion retrieval
benchmark to evaluate the robustness of our method.

##  2 Related Work

Human motion modeling. With the advances in motion capture device and pose
estimation approaches (Weinzaepfel et al. 2020; Kocabas, Athanasiou, and Black
2020; Rong, Shiratori, and Joo 2021), they enable tasks that require human
skeleton data such as motion prediction or action recognition. Since human
motion is spatio-temporal data constrained on the human body, it requires a
model to understand both temporal and spatial features. Benefiting from the
development in sequential models, there has been a significant advancement in
model-based motion models. Early studies adopted Recurrent Neural Networks
(RNN) to address various motion tasks (Fragkiadaki et al. 2015; Martinez,
Black, and Romero 2017; Wang et al. 2019; Rossi et al. 2021; Zhu et al. 2016;
Liu et al. 2016; Harvey et al. 2020). In an attempt to utilize human’s
kinematic chain, Graph Convolutional Network (GCN) also has been employed
extensively (Yan, Xiong, and Lin 2018; Shi et al. 2019; Zhang, Xu, and Tao
2020; Cheng et al. 2020a; Liu et al. 2020; Cheng et al. 2020b; Chen et al.
2021).

As the Transformer (Vaswani et al. 2017) architecture has shown remarkable
results in other domains, it has been introduced in the motion domain as well.
Specifically, as the Transformer encoder can work as a bi-directional
sequential encoder (Devlin et al. 2018), some researches (Duan et al. 2021;
Kim et al. 2022; Oreshkin et al. 2022) introduced Transformer-based motion
generation approaches. Combined with the language command, (Guo et al. 2020)
presented a RNN-based method to generate motion from language labels, and
(Petrovich, Black, and Varol 2021) used Transformer and conditional
variational autoencoder (CVAE) (Sohn, Lee, and Yan 2015) for motion generation
conditioned on language.

Joint representation learning of motion and language. Many vision and language
tasks benefit from joint training on rich visual and textual data pairs. There
are two main paradigms for using linguistic information to derive multimodal
representation, contrastive matching, and masked prediction. CLIP learns image
classification by contrastively matching images with captions (Zhang et al.
2020; Radford et al. 2021; Jia et al. 2021). The other exploit multimodal
relationships in a unified architecture. With advent of a _masked language
modeling_ objective (Devlin et al. 2018), many recent works seek to uncover
cross-modal interaction between objects and paired visual words in visual
descriptions (Tan and Bansal 2019; Li et al. 2019; Lu et al. 2019; Chen et al.
2020; Yu et al. 2020). Recently, various pivoting relationships with language
have been explored. For instance, matching video frames with visual/audio
signals to transcripts (Rouditchenko et al. 2020; Wang et al. 2021; Akbari et
al. 2021; Xu et al. 2021; Zellers et al. 2021) are broadly investigated,
thanks to the abundance resource of web videos.

Our work continues these two lines of research. We adopt a joint
representation training approach for human motion, which is relatively scarce.
We further develop our method in a distinct way beneficial to the motion
encoding domain. Our proposed MoLang enables joint representation learning
between separated domain-specific encoders. In contrast to the recent
multimodal BERT (Li et al. 2019; Lu et al. 2019; Chen et al. 2020; Yu et al.
2020), we use a separate motion encoder to keep domain-specific architecture
for motion, Graph Convolutional Bottleneck (GCB) and self-supervision
objective on motion, Masked Motion Prediction (MMP) (In section 3.1).
Following this separative domain-specific design choice, we can train motion
representation with MoLang with Contrastive Auxiliary Reconstruction (CstAR)
(In section 3.2), an auxiliary motion reconstruction objective while
contrastively matching sequence of words with masked motion. We further
investigate the design choice in ablation study (In section 4.5).

Figure 3: Joint ID and Description in skeleton configuration.

##  3 Proposed Method

In this section, we present pre-training strategy for MoLang, including:
motion representation (3.1), Masked Motion Prediction (3.1), Graph
Convolutional Bottleneck (3.1), Model architecture (3.2), and Contrastive pre-
training objectives (3.2).

###  3.1 Motion and Language Encoding

#### Motion Representation

We use 22 major joints to represent the human pose. Our skeleton is a subset
of SMPL (Loper et al. 2015), which is compatible with a large-scale motion
dataset (Mahmood et al. 2019). Skeleton configuration is depicted in Figure 3.

Each joint contains relative rotation from its parent joint. Original SMPL
parameterize poses with axis-angle rotation, but we adopted 6D representation
(Zhou et al. 2019) since it represents a rotation in the continuous domain,
which is a preferable feature for learning in neural networks. For motion
length with $T$, whole sequence can be written in $M\in\mathbb{R}^{T\times
22\times 6}$. We downsample motion if the frames per second (fps) of motion is
greater than 30Hz. If fps is less than 30 fps, motion is upsampled using
interpolation in 6D representation so that the model can consistently observe
30 fps of motion data. For the application in the downstream task, we prepend
⟨CLS⟩ token to be the starting token of motion. Therefore, input motion is
represented as $M_{\textrm{input}}\in\mathbb{R}^{(T+1)\times 22\times 6}$.

#### Masked Motion Prediction

On the basis of Transformer (Vaswani et al. 2017) encoder architecture, we
train motion encoder with masked prediction strategy (Devlin et al. 2018).
Since human motion is highly correlated with nearby frames, we mask a series
of frames instead of masking single frame in motion. We first sample the
masking motion length $l$ from uniform distribution $l\sim\mathcal{U}(1,30)$
and set the start frame of masking sequence $t_{\textrm{start}}$ from
$t_{\textrm{start}}\sim\mathcal{U}(0,T-l)$ where $T$ is the maximum length of
motion. Then, masking frames $[t_{\textrm{start}},t_{\textrm{start}+l}]$ will
be replaced by Gaussian noise to mask frames. Since motion input is
intrinsically represented in the continuous space, we do not use separate
tokenization in the motion processing pipeline. When given motion is shorter
than maximum motion length $T$, we pad remaining frames with zero and
introduce valid segmentation embedding to make the model distinguish valid
frames from padded one. This is a simple embedding of valid frames marked with
one and padded frames marked with zero. In the end, input motion, positional
embedding, and valid segmentation embedding are summed as a Transformer’s
input. We train this Masked Motion Prediction (MMP) objective on BERT for
human motion encoding and also refer to it as MMP-BERT.

Figure 4: Adapting Human body skeleton $G=\\{V:Joints,E:Bones\\}$ for human
motion sequence encoding. We put a Graph Convolutional Bottleneck, (GCB)
between Multi-Head Self-Attention (MHSA) Blocks (Vaswani et al. 2017) in our
motion encoder, MMP-BERT.

#### Graph Convolutional Bottleneck

The human body can be interpreted as a graph structure. We can consider
(Joints, Bones) in the human body as $G=\\{\textrm{Nodes},\textrm{Edges}\\}$.
Node is denoted as $\mathcal{J}=\\{j_{1},j_{2},\cdots,j_{J}\\}$, where $J$ is
the number of joints. Adjacency matrix
$\mathbf{A}=(a_{ik})\in\mathbb{R}^{J\times J}$ represent correlation between
joints $j_{i}$ and $j_{k}$. Neighboring joints of $j_{i}$ is denoted as:

$\mathcal{N}(j_{i})=\left\\{j_{k}\mid a_{ik}\neq 0\right\\}$ (1)

We define the elements of the adjacency matrix to have one if joints are
connected and zero otherwise. For the feature dimension of $d$, the output of
$L$-th encoder layer $\mathbf{H}^{L}\in\mathbb{R}^{J\times d}$ is computed in
the graph convolutional layer as:

$\mathbf{y}_{i}=\sigma\left(\sum_{j_{k}\in\mathcal{N}(j_{i})}a_{ik}\mathbf{h}^{L}_{k}\mathbf{W}\right)$
(2)

where $\mathbf{W}\in\mathbb{R}^{d\times d}$ is a trainable weight and $\sigma$
is a non-linear activation function. We insert the graph convolutional layer
between Multi-Head Self-Attention Blocks (Vaswani et al. 2017) to prioritize
relation in the group of neighboring joints. Our motion encoder architecture
is shown in Figure 4.

#### Language Encoding

We use BERT (Devlin et al. 2018) architecture as our text encoder. Instead of
training from scratch, we initialize the text encoder with ‘bert-base-uncased’
pre-trained weights provided from HuggingFace Transformers (Wolf et al. 2020).
Language transformer is also finetuned in our contrastive learning pipeline.

###  3.2 Joint Model Architecture

Figure 5: An overview of MoLang training pipeline. We provide sequence-level
representations of human motion to a Motion encoder. The motion encoder
contextualizes over fine-grained motion sequence with Masked Motion Prediction
objective and further guided with Symmetric Cross-Entropy by pre-trained
Language encoder. The model must maximize its similarity of language label and
motion encoding while keeping it contextualized.

An overview of MoLang is shown in Figure 5. MoLang is trained on the top of
the pre-trained motion encoder (MMP-BERT) discussed in 3.1. Each motion and
language modality is first encoded separately by a modality-specific pre-
trained transformer encoder. Then, the representations will be trained with
contrastive loss. Our model use transformer architecture as motion encoder and
contrastive loss with Auxiliary Reconstruction loss.

#### Contrastive learning with Auxiliary Reconstruction (CstAR)

We use contrastive learning with the InfoNCE loss (Sohn 2016; Van den Oord,
Li, and Vinyals 2018) with auxiliary motion reconstruction loss
$\mathcal{L}_{\textrm{recon}}$. Auxiliary motion reconstruction loss is an
L1-norm between the output of the Motion encoder and input motion. The
auxiliary term makes our motion encoder predict masked motion while adapting
cross-modal matching (i.e., contrastive learning) between motion and language
domain. We optimize the CstAR loss:

$\displaystyle\mathcal{L}_{CstAR}(M,L)=$
$\displaystyle\\!\\!\\!\sum_{i}\left(\frac{\exp{\left(s(\bm{m}^{(i)},\bm{l}^{(i)})/\tau\right)}}{\sum_{\bm{m}}\exp{\left(s(\bm{m},\bm{l}^{(i)})/\tau\right)}}+\frac{\exp{\left(s(\bm{m}^{(i)},\bm{l}^{(i)})/\tau\right)}}{\sum_{\bm{l}}\exp{\left(s(\bm{m}^{(i)},\bm{l})/\tau\right)}}+\alpha\mathcal{L}_{\textrm{recon}}\right)$
$\displaystyle\mbox{where}\;\;\mathcal{L}_{\textrm{recon}}=\left\lVert\bm{m}^{(i)}-\bm{m}^{(i)}_{\textrm{input}}\right\rVert_{1}$
(3)

where $\tau$ is a learnable temperature, $\alpha$ is a weight for auxiliary
motion reconstruction loss, and $\bm{m}^{(i)},\bm{l}^{(i)}$ are vector
representations of the co-occuring motion-language pair $(m^{(i)},l^{(i)})$
which are encoded by $g_{M}(m^{(i)})$ and Language encoder $g_{L}(l^{(i)})$,
Motion encoder respectively; $s(*,*)$ computes the scaled cosine similarity.
Vector similarities across modalities are optimized to reconstruct co-
occurrence patterns in training corpora.

##  4 Experiments

This section describes the datasets and experiment settings and demonstrates
the experimental results of our model.

###  4.1 Datasets

The following datasets are used in our experiment. All datasets used in our
experiments follows SMPL (Loper et al. 2015) representation format. Unlike the
3D poses from AMASS (Mahmood et al. 2019) and BABEL (Punnakkal et al. 2021),
which are obtained from accurate motion capture equipment, 3D pose of other
datasets are obtained by pose estimation framework such as SMPLify (Bogo et
al. 2016) or VIBE (Kocabas, Athanasiou, and Black 2020).

AMASS (Mahmood et al. 2019) This is a large collection of motion dataset.
Datasets collected in this dataset are obtained from a highly accurate optical
marker-based motion capture system. It integrates more than 50-hours mocap
datasets by using widely used SMPL format.

BABEL (Punnakkal et al. 2021) This is a language annotation for the AMASS
dataset. Unlike most previous motion labels, which only describe motions with
high-level semantic labels, this provides frame-level annotation. Also, it
describes motion closer to natural language rather than using single-word
categorical labels.

HumanAct12 (Guo et al. 2020) HumanAct12 dataset is a annotated version of
PHSPD (Zou et al. 2020) dataset with temporal cropping. In order to obatin 3D
poses, it employed multi-step fitting by OpenPose (Cao et al. 2017) and
SMPLify (Bogo et al. 2016). It comprises 1,191 motions with 12 labels.

Refined NTU-RGBD 13 (Guo et al. 2020) This dataset is a subset of NTU-RGBD
(Shahroudy et al. 2016) action recognition dataset, published by (Guo et al.
2020). It is a re-estimationion of 3D poses through VIBE (Kocabas, Athanasiou,
and Black 2020) to suppress noise in original NTU-RGBD dataset. This comprises
3,902 motion clips with 13 action classes.

UESTC (Ji et al. 2019) This dataset consists of 40 aerobic exercise
categories. VIBE (Kocabas, Athanasiou, and Black 2020) is employed to estimate
SMPL parameters from the original dataset. We split train and test by cross-
subject recognition protocol, motions recorded from 51 subjects for the train
split and the others for the test split.

###  4.2 Implementation Details

#### Preprocessing

Since motion datasets have various frame rates, we make every motion have the
same (30 fps) frame rates. Interpolation is conducted in 6D continuous
representation (Zhou et al. 2019). The maximum length of motion is set to 150
frames (5 seconds). The preprocessed 364K samples in AMASS (Mahmood et al.
2019) are used to pre-train the motion encoder.

In the BABEL dataset, we use frame-level processed annotation for contrastive
learning. We exclude motions annotated as ‘transition’ since it does not
convey consistent semantic information for describing motion. Unique language
descriptions used in the training set are about five thousand.

#### Hyperparameters

Our motion encoder is a ten-layer transformer encoder with 12 heads and 1,024
feedforward dimensions. Graph convolutional bottleneck is located between the
fourth and fifth transformer encoder layers. The maximum context length is set
to 150. In the contrastive training setting, the output of the motion encoder
is projected to 768 dimensions to match with the text encoder. To process text
inputs, we use pre-trained BERT (Devlin et al. 2018). Learnable temperature is
used to compute symmetric cross-entropy (SCE), initialized with 0.07. Weights
for $\mathcal{L}_{\textrm{recon}}$ is set to 10.0.

#### Training

Our model takes two training stages. First, our model is trained in a self-
supervised manner with motion capture datasets (i.e. AMASS (Mahmood et al.
2019)). In the second stage, MoLang is trained on BABEL motion-language paired
training dataset together using CstAR.

###  4.3 Sekelton-based Action Recognition

To validate the effectiveness of our motion representation, we evaluate our
model on skeleton-based action recognition downstream tasks. As our text
encoder can process natural languages, we directly input the class label of
the target dataset to our text encoder, which is an advantage over other
classifiers that are fixed to class labels at the time of training. In this
transfer learning setting, we finetune on target dataset with the same
contrastive objective. We evaluate our approach on the HumanAct12, Refined
NTURGBD13, and UESTC in Table 1.

Methods |  HumanAct12 |  NTU-RGBD13 |  UESTC
---|---|---|---
LSTM |  68.07 |  80.58 |  78.33
P-LSTM (Shahroudy et al. 2016) |  70.20 |  82.03 |  80.14
TCN (Lea et al. 2017) |  75.21 |  84.54 |  85.77
2s-AGCN (Shi et al. 2019) |  79.01 |  90.63 |  90.86
2s-AGCN (Shi et al. 2019) (pretrain) |  83.19 |  91.67 |  95.63
DC-GCN (Cheng et al. 2020a) |  82.72 |  90.75 |  90.51
DC-GCN (Cheng et al. 2020a) (pretrain) |  85.46 |  92.71 |  96.58
MMP-BERT |  84.87 |  85.77 |  90.76
MMP-BERT (pretrain) |  88.82 |  91.85 |  94.74
MoLang |  91.48 |  93.86 |  96.97
Table 1: Comparison of accuracy on skeleton-based action recognition task.
Pre-trained models are trained on BABEL (Punnakkal et al. 2021) dataset.
Figure 6: Qualitative results on action recognition downstream tasks,
HumanAct12 and UESTC.

Since GCN-based action recognition models take $xyz$-coordinates in Euclidean
space, we convert our 6D representation to joint positions using (Loper et al.
2015) to make a fair comparison for the baseline models. Both Deep LSTM and
P-LStM (Shahroudy et al. 2016) is based on LSTM architecture and used to
evaluate action recognition in large-scale RGB+D action recognition dataset
(Shahroudy et al. 2016). Deep LSTM is traditional RNN-based architecture, and
P-LSTM is a part-aware LSTM model, which splits the human body’s motion into
predefined parts. TCN (Lea et al. 2017) is a multi-layer 1-dimensional
convolution network, which takes convolution operation on the temporal
dimension. The 2s-AGCN (Shi et al. 2019) is a graph convolution-based
architecture that conducts convolutions on both spatial and temporal
dimensions with two streams of networks. Decouple GCN (DC-GCN) (Cheng et al.
2020a) break channels into several groups to enhance spatial aggregation. MMP-
BERT is a Transformer encoder layer trained with our masked motion prediction
strategy. The pre-trained version is trained on BABEL (Punnakkal et al. 2021)
dataset before downstream tasks.

Figure 7: Qualitative results on motion representation. The closest three
language descriptions are chosen from BABEL datset.

To evaluate the capability of language description, we also test our model on
a separate set of BABEL for qualitative evaluation. We give more than five
thousand labels that depict motion to the text encoder and compare the labels
with the top 3 similarities with motion. This is shown in Figure 7.

$MMP$ |  $GCB$ |  $CstAR$ |  Human Act12 |  NTU-RGBD13 |  UESTC
---|---|---|---|---|---
$-$ |  $-$ |  $-$ |  82.35 |  88.02 |  91.79
$\checkmark$ |  $-$ |  $-$ |  90.31 |  92.51 |  95.26
$-$ |  $\checkmark$ |  $-$ |  84.2 |  88.84 |  91.93
$-$ |  $-$ |  $\checkmark$ |  82.65 |  88.75 |  92.73
$\checkmark$ |  $\checkmark$ |  $-$ |  90.33 |  92.53 |  95.39
$\checkmark$ |  $-$ |  $\checkmark$ |  91.07 |  92.67 |  95.37
$\checkmark$ |  $\checkmark$ |  $\checkmark$ |  91.48 |  93.86 |  96.97
Table 2: (Action recognition) Ablation study on self-supervised pre-training
of Masked Motion Prediction (MMP), Graph Convolutional Bottleneck (GCB), and
Auxiliary Reconstruction loss (CstAR).

###  4.4 Motion Retrieval

Figure 8: Qualitative results on motion retrieval. The closest motion for
given label is retrieved.

We make multiple choice motion retrieval questions from the BABEL dataset for
the motion retrieval task. First, we choose frequent and visually exclusive 15
labels (e.g., sit, dance, wipe, crouch, etc.) from the entire labels. Then,
motion is categorized by those labels to compose candidates for each label. At
the evaluation phase, the model is asked to find the closest motion for the
given text among 15 randomly chosen motions. We compose 450 motion retrieval
questions to evaluate motion retrieval performance. Qualitative results are
presented in Figure 8.

###  4.5 Ablation study

We also conduct an ablation study to empirically validate our self-supervised
pre-training design decisions on motion, Graph Convolutional Bottleneck, and
Auxiliary Reconstruction loss for contrastive learning (CstAR).

Results are in Table 2 (Sekelton-based Action Recognition) and Table 3 (Motion
Retrieval)

Motion-only self-supervised pre-training helps contrastive learning. We pre-
train our motion encoder in a self-supervised fashion before contrastive pre-
training, and we call this MMP pre-training here. (details in section 3.1) MMP
pre-training always helps in downstream and retrieval when preceded
beforehand. This confirms pre-training power of domain encoder has an
essential role in robust contrastive learning.

GCB helps motion representation. We employ an adjacency matrix-based graph
convolutional layer to make an information bottleneck, which encourages the
model to focus on neighboring joints over distant ones. Specifically, combined
with CstAR, GCB significantly improves model performance. (6th row vs 7th row)
We assume that the model learns meaningful priors for the understanding
sequence of human poses from using GCB.

CstAR, Auxiliary reconstruction loss helps robust cross-modal matching.
Comparing (MMP \+ GCB, 5th row) with full model (7th row) in Table 2,3, we see
the accuracy increases in both classification and retrieval. Even simple
contrastive learning (1st row) improves with the $CstAR$ objective (4th row).
We believe two advantages can explain the improvements. One reason is, The
loss helps the model to pay attention on motion completion when updating for
contrastive learning. The other reason stems from the motion augmentation
effect during contrastive learning. This also confirms that the auxiliary
reconstruction loss for masked motion guides robust contrastive matching
during motion-language alignment.

$MMP$ |  $GCB$ |  $CstAR$ |  Top-1 (%) |  Top-3 (%)
---|---|---|---|---
$-$ |  $-$ |  $-$ |  48.44 |  74.22
$\checkmark$ |  $-$ |  $-$ |  55.18 |  75.14
$-$ |  $\checkmark$ |  $-$ |  49.64 |  74.18
$-$ |  $-$ |  $\checkmark$ |  50.24 |  74.39
$\checkmark$ |  $\checkmark$ |  $-$ |  55.06 |  75.25
$\checkmark$ |  $-$ |  $\checkmark$ |  56.12 |  76.62
$\checkmark$ |  $\checkmark$ |  $\checkmark$ |  56.76 |  77.34
Table 3: Ablation study on self-supervised pre-training of Masked Motion
Prediction (MMP), Graph Convolutional Bottleneck (GCB), and Auxiliary
Reconstruction loss (CstAR). Top-1 and Top-3 accuracy on motion retrieval
task.

##  5 Conclusion

We introduce MoLang, which learns self-supervised representations jointly
through motion and language modalities. We have found empirically that our
model achieves robust representation in motion-language matching. Furthermore,
we strengthen our contrastive matching objective by utilizing auxiliary
reconstruction loss. Our model performs well in both downstream action
recognition benchmarks and motion-text retrieval settings.

Moreover, the motion-language similarity evaluation metric, available from our
MoLang, is an essential foundation for leveraging human motion-language pairs
in the wild, such as web videos and movie films. Specifically, curating and
incorporating web-scale multimodal data (e.g., pairs of estimated human motion
and noisy subtitles) for effective human motion understanding is a promising
direction and one that has significant space for future work.

## References

  * Akbari et al. (2021) Akbari, H.; Yuan, L.; Qian, R.; Chuang, W.-H.; Chang, S.-F.; Cui, Y.; and Gong, B. 2021.  VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text.  _arXiv preprint arXiv:2104.11178_.
  * Bogo et al. (2016) Bogo, F.; Kanazawa, A.; Lassner, C.; Gehler, P.; Romero, J.; and Black, M. J. 2016\.  Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image.  In _Computer Vision – ECCV 2016_ , Lecture Notes in Computer Science. Springer International Publishing.
  * Cao et al. (2017) Cao, Z.; Simon, T.; Wei, S.-E.; and Sheikh, Y. 2017.  Realtime multi-person 2d pose estimation using part affinity fields.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 7291–7299.
  * Chen et al. (2021) Chen, Y.; Zhang, Z.; Yuan, C.; Li, B.; Deng, Y.; and Hu, W. 2021.  Channel-wise topology refinement graph convolution for skeleton-based action recognition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 13359–13368.
  * Chen et al. (2020) Chen, Y.-C.; Li, L.; Yu, L.; El Kholy, A.; Ahmed, F.; Gan, Z.; Cheng, Y.; and Liu, J. 2020.  Uniter: Universal image-text representation learning.  In _European conference on computer vision_ , 104–120. Springer.
  * Cheng et al. (2020a) Cheng, K.; Zhang, Y.; Cao, C.; Shi, L.; Cheng, J.; and Lu, H. 2020a.  Decoupling gcn with dropgraph module for skeleton-based action recognition.  In _European Conference on Computer Vision_ , 536–553. Springer.
  * Cheng et al. (2020b) Cheng, K.; Zhang, Y.; He, X.; Chen, W.; Cheng, J.; and Lu, H. 2020b.  Skeleton-based action recognition with shift graph convolutional network.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 183–192.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv preprint arXiv:1810.04805_.
  * Duan et al. (2021) Duan, Y.; Shi, T.; Zou, Z.; Lin, Y.; Qian, Z.; Zhang, B.; and Yuan, Y. 2021.  Single-shot motion completion with transformer.  _arXiv preprint arXiv:2103.00776_.
  * Fragkiadaki et al. (2015) Fragkiadaki, K.; Levine, S.; Felsen, P.; and Malik, J. 2015.  Recurrent network models for human dynamics.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 4346–4354.
  * Guo et al. (2020) Guo, C.; Zuo, X.; Wang, S.; Zou, S.; Sun, Q.; Deng, A.; Gong, M.; and Cheng, L. 2020\.  Action2motion: Conditioned generation of 3d human motions.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , 2021–2029.
  * Harvey et al. (2020) Harvey, F. G.; Yurick, M.; Nowrouzezahrai, D.; and Pal, C. 2020.  Robust motion in-betweening.  _ACM Transactions on Graphics (TOG)_ , 39(4): 60–1.
  * Ji et al. (2019) Ji, Y.; Xu, F.; Yang, Y.; Shen, F.; Shen, H. T.; and Zheng, W.-S. 2019.  A large-scale varying-view RGB-D action dataset for arbitrary-view human action recognition.  _arXiv preprint arXiv:1904.10681_.
  * Jia et al. (2021) Jia, C.; Yang, Y.; Xia, Y.; Chen, Y.-T.; Parekh, Z.; Pham, H.; Le, Q. V.; Sung, Y.; Li, Z.; and Duerig, T. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  _arXiv preprint arXiv:2102.05918_.
  * Kim et al. (2022) Kim, J.; Byun, T.; Shin, S.; Won, J.; and Choi, S. 2022.  Conditional Motion In-betweening.  _arXiv preprint arXiv:2202.04307_.
  * Kocabas, Athanasiou, and Black (2020) Kocabas, M.; Athanasiou, N.; and Black, M. J. 2020.  Vibe: Video inference for human body pose and shape estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5253–5263.
  * Lea et al. (2017) Lea, C.; Flynn, M. D.; Vidal, R.; Reiter, A.; and Hager, G. D. 2017.  Temporal convolutional networks for action segmentation and detection.  In _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 156–165.
  * Li et al. (2019) Li, L. H.; Yatskar, M.; Yin, D.; Hsieh, C.-J.; and Chang, K.-W. 2019.  Visualbert: A simple and performant baseline for vision and language.  _arXiv preprint arXiv:1908.03557_.
  * Liu et al. (2016) Liu, J.; Shahroudy, A.; Xu, D.; and Wang, G. 2016.  Spatio-temporal lstm with trust gates for 3d human action recognition.  In _European conference on computer vision_ , 816–833. Springer.
  * Liu et al. (2020) Liu, Z.; Zhang, H.; Chen, Z.; Wang, Z.; and Ouyang, W. 2020.  Disentangling and unifying graph convolutions for skeleton-based action recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 143–152.
  * Loper et al. (2015) Loper, M.; Mahmood, N.; Romero, J.; Pons-Moll, G.; and Black, M. J. 2015.  SMPL: A skinned multi-person linear model.  _ACM transactions on graphics (TOG)_ , 34(6): 1–16.
  * Lu et al. (2019) Lu, J.; Batra, D.; Parikh, D.; and Lee, S. 2019.  ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _Advances in Neural Information Processing Systems_ , 13–23.
  * Mahmood et al. (2019) Mahmood, N.; Ghorbani, N.; Troje, N. F.; Pons-Moll, G.; and Black, M. J. 2019.  AMASS: Archive of Motion Capture as Surface Shapes.  In _International Conference on Computer Vision_ , 5442–5451.
  * Marszałek, Laptev, and Schmid (2009) Marszałek, M.; Laptev, I.; and Schmid, C. 2009.  Actions in Context.  In _IEEE Conference on Computer Vision & Pattern Recognition_.
  * Martinez, Black, and Romero (2017) Martinez, J.; Black, M. J.; and Romero, J. 2017.  On human motion prediction using recurrent neural networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2891–2900.
  * Miech et al. (2019) Miech, A.; Zhukov, D.; Alayrac, J.-B.; Tapaswi, M.; Laptev, I.; and Sivic, J. 2019\.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2630–2640.
  * Oreshkin et al. (2022) Oreshkin, B. N.; Valkanas, A.; Harvey, F. G.; Ménard, L.-S.; Bocquelet, F.; and Coates, M. J. 2022.  Motion Inbetweening via Deep Delta-Interpolator.  _arXiv preprint arXiv:2201.06701_.
  * Petrovich, Black, and Varol (2021) Petrovich, M.; Black, M. J.; and Varol, G. 2021.  Action-conditioned 3d human motion synthesis with transformer vae.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 10985–10995.
  * Punnakkal et al. (2021) Punnakkal, A. R.; Chandrasekaran, A.; Athanasiou, N.; Quiros-Ramirez, A.; and Black, M. J. 2021.  BABEL: Bodies, Action and Behavior with English Labels.  In _Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 722–731.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  _arXiv preprint arXiv:2103.00020_.
  * Rohrbach et al. (2016) Rohrbach, A.; Torabi, A.; Rohrbach, M.; Tandon, N.; Chris, P.; Hugo, L.; Aaron, C.; and Schiele, B. 2016.  Movie Description.  _arXiv preprint_.
  * Rong, Shiratori, and Joo (2021) Rong, Y.; Shiratori, T.; and Joo, H. 2021.  Frankmocap: A monocular 3d whole-body pose estimation system via regression and integration.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 1749–1759.
  * Rossi et al. (2021) Rossi, L.; Paolanti, M.; Pierdicca, R.; and Frontoni, E. 2021.  Human trajectory prediction and generation using LSTM models and GANs.  _Pattern Recognition_ , 120: 108136.
  * Rouditchenko et al. (2020) Rouditchenko, A.; Boggust, A.; Harwath, D.; Chen, B.; Joshi, D.; Thomas, S.; Audhkhasi, K.; Kuehne, H.; Panda, R.; Feris, R.; et al. 2020.  Avlnet: Learning audio-visual language representations from instructional videos.  _arXiv preprint arXiv:2006.09199_.
  * Shahroudy et al. (2016) Shahroudy, A.; Liu, J.; Ng, T.-T.; and Wang, G. 2016.  Ntu rgb+ d: A large scale dataset for 3d human activity analysis.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1010–1019.
  * Shi et al. (2019) Shi, L.; Zhang, Y.; Cheng, J.; and Lu, H. 2019.  Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition.  In _CVPR_.
  * Sigurdsson, Russakovsky, and Gupta (2017) Sigurdsson, G. A.; Russakovsky, O.; and Gupta, A. K. 2017.  What Actions are Needed for Understanding Human Actions in Videos?  _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2156–2165.
  * Sigurdsson et al. (2016) Sigurdsson, G. A.; Varol, G.; Wang, X.; Farhadi, A.; Laptev, I.; and Gupta, A. K. 2016.  Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding.  _ArXiv_ , abs/1604.01753.
  * Sohn (2016) Sohn, K. 2016.  Improved Deep Metric Learning with Multi-class N-pair Loss Objective.  In Lee, D.; Sugiyama, M.; Luxburg, U.; Guyon, I.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 29. Curran Associates, Inc.
  * Sohn, Lee, and Yan (2015) Sohn, K.; Lee, H.; and Yan, X. 2015.  Learning structured output representation using deep conditional generative models.  _Advances in neural information processing systems_ , 28.
  * Tan and Bansal (2019) Tan, H.; and Bansal, M. 2019.  LXMERT: Learning cross-modality encoder representations from transformers.  In _EMNLP_.
  * Van den Oord, Li, and Vinyals (2018) Van den Oord, A.; Li, Y.; and Vinyals, O. 2018.  Representation learning with contrastive predictive coding.  _arXiv e-prints_ , arXiv–1807.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2019) Wang, H.; Ho, E. S.; Shum, H. P.; and Zhu, Z. 2019.  Spatio-temporal manifold learning for human motions via long-horizon modeling.  _IEEE transactions on visualization and computer graphics_ , 27(1): 216–227.
  * Wang et al. (2021) Wang, L.; Luc, P.; Recasens, A.; Alayrac, J.-B.; and Oord, A. v. d. 2021.  Multimodal Self-Supervised Learning of General Audio Representations.  _arXiv preprint arXiv:2104.12807_.
  * Weinzaepfel et al. (2020) Weinzaepfel, P.; Brégier, R.; Combaluzier, H.; Leroy, V.; and Rogez, G. 2020\.  Dope: Distillation of part experts for whole-body 3d pose estimation in the wild.  In _European Conference on Computer Vision_ , 380–397. Springer.
  * Wolf et al. (2020) Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Scao, T. L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. M. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 38–45. Online: Association for Computational Linguistics.
  * Xu et al. (2021) Xu, H.; Ghosh, G.; Huang, P.-Y.; Okhonko, D.; Aghajanyan, A.; and Feichtenhofer, F. M. L. Z. C. 2021.  VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding.  _arXiv preprint arXiv:2109.14084_.
  * Yan, Xiong, and Lin (2018) Yan, S.; Xiong, Y.; and Lin, D. 2018.  Spatial temporal graph convolutional networks for skeleton-based action recognition.  In _Thirty-second AAAI conference on artificial intelligence_.
  * Yu et al. (2020) Yu, F.; Tang, J.; Yin, W.; Sun, Y.; Tian, H.; Wu, H.; and Wang, H. 2020.  ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph.  _arXiv preprint arXiv:2006.16934_.
  * Zellers et al. (2021) Zellers, R.; Lu, X.; Hessel, J.; Yu, Y.; Park, J. S.; Cao, J.; Farhadi, A.; and Choi, Y. 2021.  Merlot: Multimodal neural script knowledge models.  _Advances in Neural Information Processing Systems_ , 34.
  * Zhang, Xu, and Tao (2020) Zhang, X.; Xu, C.; and Tao, D. 2020.  Context aware graph convolution for skeleton-based action recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 14333–14342.
  * Zhang et al. (2020) Zhang, Y.; Jiang, H.; Miura, Y.; Manning, C. D.; and Langlotz, C. P. 2020.  Contrastive learning of medical visual representations from paired images and text.  _arXiv preprint arXiv:2010.00747_.
  * Zhou et al. (2019) Zhou, Y.; Barnes, C.; Lu, J.; Yang, J.; and Li, H. 2019.  On the continuity of rotation representations in neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5745–5753.
  * Zhu et al. (2016) Zhu, W.; Lan, C.; Xing, J.; Zeng, W.; Li, Y.; Shen, L.; and Xie, X. 2016.  Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 30.
  * Zou et al. (2020) Zou, S.; Zuo, X.; Qian, Y.; Wang, S.; Xu, C.; Gong, M.; and Cheng, L. 2020.  3D human shape reconstruction from a polarization image.  In _European Conference on Computer Vision_ , 351–368. Springer.

##  Appendix A Appendix

We present the following items in the supplemental:

  1. 1.

More Qualitative Results

  2. 2.

t-SNE Visualization of Language Representations

  3. 3.

Hyperparameters

##  Appendix B Appendix 1: More Qualitative Results

###  B.1 More Skeleton-based Action Recognition Results

More action recognition examples in four datasets are provided in Figure 9,
Figure 10, Figure 11, and Figure 12.

Figure 9: Action recognition for BABEL dataset. Top-3 predicted labels from
BABEL dataset. Figure 10: Action recognition for HumanAct12 dataset. Top-3
predicted labels from BABEL dataset. Figure 11: Action recognition for NTU-
RGBD 13 dataset. Top-3 predicted labels from BABEL dataset. Figure 12: Action
recognition for UESTC dataset. Top-3 predicted labels from BABEL dataset.

###  B.2 More Motion Retrieval Results

An additional motion retrieval example is provided in Figure 13.

Figure 13: Motion retrieval from BABEL dataset.

##  Appendix C Appendix 2: t-SNE Visualization of Language Representations

Visualizations of latent space of text encoder in t-SNE. Figure 14, Figure 15,
Figure 16, and Figure 17 show better understanding of the relation between
motion and text rather than the original text encoder.

Figure 14: t-SNE results from text encoder. (yellow): motion using legs.
(blue): motion using arms. Figure 15: t-SNE results from text encoder.
(yellow): motion using knees. (pink): walking motion. (orange): body rotating
motion. (purple): jumping motion. Figure 16: t-SNE results from text encoder.
(yellow): throwing motion. (pink): motion that move arms near chest. (orange):
object interaction using hands. (purple): motion of moving arms near head.
Figure 17: t-SNE results from text encoder. (yellow): motion using legs.
(blue): motion using arms. (pink): motion using legs.

##  Appendix D Appendix 3: Hyperparameters Experimental Setup

Hyperparameters of motion encoder and MoLang are listed in Table 4 and Table
5.

Hyperparameter | Motion Encoder
---|---
Optimizer | Adam
Learning Rate | 0.0001
$\beta_{1}$ | 0.9
$\beta_{2}$ | 0.999
Scheduler | Cosine Annealing
$T_{0}$ | 20
$T_{\textrm{mult}}$ | 1
$\eta_{\textrm{min}}$ | 0.00007
Batch Size | 512
Num Layers | 10
Heads | 12
Maximum Length | 150
Feedforward Dimension | 1024
Dropout Rate | 0.1
Embedding Dimension | 768
GCB Layer | between 4th and 5th Layer
Table 4: Hyperparameters for training motion encoder Hyperparameter | MoLang
---|---
Optimizer | Adam
Learning Rate | 0.0001
$\beta_{1}$ | 0.9
$\beta_{2}$ | 0.999
Scheduler | Cosine Annealing
$T_{0}$ | 20
$T_{\textrm{mult}}$ | 1
$\eta_{\textrm{min}}$ | 0.00007
Text Encoder | Pretrained BERT (‘bert-base-uncased‘)
Temperature | 0.07
Weight for $\mathcal{L}_{\textrm{recon}}$ | 10
Table 5: Hyperparameters for training MoLang