# MotionLLM: Multimodal Motion-Language Learning with Large Language Models

Qi Wu
HKUST
<EMAIL_ADDRESS>
&Yubo Zhao11footnotemark: 1
HKUST
<EMAIL_ADDRESS>
&Yifan Wang11footnotemark: 1
HKUST
<EMAIL_ADDRESS>
&Yu-Wing Tai
Dartmouth College
<EMAIL_ADDRESS>
&Chi-Keung Tang
HKUST
<EMAIL_ADDRESS>
Equal contribution.

###### Abstract

Recent advancements in Multimodal Large Language Models (MM-LLMs) have
demonstrated promising potential in terms of generalization and robustness
when applied to different modalities. While previous works have already
achieved 3D human motion generation using various approaches including
language modeling, they mostly use specialized architecture and are restricted
to single-human motion generation. Inspired by the success of MM-LLMs, we
propose MotionLLM, a simple and general framework that can achieve single-
human, multi-human motion generation, and motion captioning by fine-tuning
pre-trained LLMs. Specifically, we encode and quantize motions into discrete
LLM-understandable tokens, which results in a unified vocabulary consisting of
both motion and text tokens. With only 1–3% parameters of the LLMs trained by
using adapters, our single-human motion generation achieves comparable results
to those diffusion models and other trained-from-scratch transformer-based
models. Additionally, we show that our approach is scalable and flexible,
allowing easy extension to multi-human motion generation through
autoregressive generation of single-human motions. Project page:
https://knoxzhao.github.io/MotionLLM

##  1 Introduction

Large Language Models (LLMs) have attracted great attention in both industry
and academia. Many LLMs, such as GPT-4 [1], LLaMA [35], Gemma [32], have shown
their advanced capabilities in robustness and generalization across various
downstream tasks. These progresses have motivated researchers to explore the
application of LLMs in multimodal tasks, such as integrating them with
modalities like images [18], videos [44], audio [2, 14], and more, resulting
in promising outcomes. However, the utilization of LLMs in the context of 3D
human motion remains an underexplored area of research.

Research in 3D human motion has explored various language-related tasks,
including motion generation [45, 9, 34, 46, 31, 8, 16] (text to motion) and
motion captioning [10, 16] (motion to text). These tasks essentially involve
establishing a translation mapping between language and motion. Many previous
approaches such as [45, 8, 34], have primarily relied on pre-trained text
encoders, such as CLIP [30], to bridge the gap between text and motion.
Methods such as [10] model the correlation between text and motion. However,
these methods may encounter difficulties in handling semantically complex
texts due to limited language understanding and face challenges in achieving
bidirectional translation. Furthermore, most existing approaches employ task-
specific architectures that lack adaptability to different tasks and may
struggle when presented with unseen data.

In our work, we propose MotionLLM, a simple yet effective motion-language
learning framework that leverages the remarkable language understanding,
generation abilities, robustness, and flexibility of pre-trained LLMs. More
specifically, we train a pair of motion tokenizer and detokenizer. The motion
tokenizer encodes motions into motion embeddings and quantizes them into a set
of discrete LLM-understandable tokens using a codebook, while the detokenizer
reconstructs tokens back to their original continuous forms. This tokenizer-
detokenizer pair enables the translation between continuous motion sequences
and discrete tokens, facilitating interaction with the LLM while still
allowing for the recovery of the original motions from the tokens. We enrich
the LLM’s vocabulary with these additional motion tokens, while keeping the
text tokens unchanged, to form an enhanced motion-language vocabulary. Given
that motions can be represented as temporal sequences, our tokenization
process translates motions into token sequences akin to sentences in natural
language. In terms of the LLM, it is always token-in-token-out, allowing
different tasks to be accomplished by simply specifying distinct token inputs
and outputs, for fully leveraging the inherent versatility of LLMs.

The utilization of LLMs in our approach presents several advantages. By fine-
tuning a pre-trained, lightweight LLM using adapters, we achieve competitive
results in single-human motion generation compared to those trained-from-
scratch models with specialized architectures. Our MotionLLM exhibits an
enhanced comprehension of intricate text descriptions, enabling the
accommodation of diverse and customized language inputs. Notably, our model
can generate motions based on unseen language descriptions. In motion
captioning, we achieve state-of-the-art performances, generating semantically
accurate and contextually appropriate text descriptions. Our model outperforms
existing models by effectively capturing the essence of motion with clear
motivation, and providing accurate descriptions when the motion underlies more
subtlety and is less obvious. Furthermore, we also demonstrate our method’s
strong ability of generalization and flexibility by extending to more complex
multi-human motion generation, which remains an underexplored field, through
an easy adaptation of single-human motion generation. To our best knowledge,
our method is the first to use discrete tokens for multi-human motion
generation. These achievements are realized through a simple training
pipeline, training only 1-3% parameters of the LLM using adapters. This
highlights the efficiency and effectiveness of our MotionLLM in harnessing the
power of LLMs for 3D human motion-language tasks.

To summarize, our contributions include:

  * •

We introduce a simple yet effective motion-language learning framework that
effectively utilizes LLMs and can be easily adapted to various motion-language
tasks.

  * •

We show that, by only fine-tuning a lightweight LLM with a small number of
trainable parameters (1-3%) using adapters, our MotionLLM yields satisfactory
results in different motion-language tasks.

  * •

We demonstrate the flexibility of our method by readily extending MotionLLM to
generate multi-human motions.

##  2 Related Work

Multimodal LLMs Recent advancements have integrated large language models
(LLMs) with multiple modalities such as image, video, music, audio, and point
cloud [22, 12, 40, 3, 6] using different approaches. Various approaches have
been proposed to align different modalities. For instance, Video-LLaMA [44]
leverages Q-formers to bridge the gap between modalities. PointLLM [43]
utilizes a projector to align the feature space of point clouds with the
feature space of the LLM. VALLE-X [50] tokenizes audio inputs to establish a
connection between audio and language modalities. For 3D motions, MotionGPT
[16] adopts a language modeling approach by treating single-human motions as a
foreign language with additional trained vocabulary. However, it does not
fully exploit the advantages of LLMs and relies on massive pre-training. Our
investigation demonstrates that simply a pre-trained LLM with an adapter is
already sufficient for such multimodal understanding, eliminating the need for
extensive pre-training. And thanks to the linguistic capabilities of pre-
trained LLMs, our method demonstrates substantial improvements in language
comprehension. Furthermore, our paper contributes a scalable approach to
generate multi-human 3D motions. Unlike [20] where extension beyond 2-person
motions is non-trivial, by capitalizing on LLM’s autoregressive generation,
this paper shows preliminary and encouraging results on text-guided multi-
human motion generation.

3D Human Motion Generation Modern works can generate human motions based on a
variety of inputs such as action labels [26, 19, 11, 42], textual descriptions
[16, 39, 45, 10, 51, 34, 33, 8, 46, 47, 5, 27, 49], control signals [41, 38,
28, 15, 7], music or audio [5, 36, 52], and others [17]. Particularly, text-
guided 3D motion generation or text-to-motion has garnered significant
interest. Notably, some diffusion models have emerged as powerful tools for
single-human motion [34, 31, 39, 51, 41, 46] and multi-human motion generation
[20, 31] using texts. Despite the proficiency in generating motions, diffusion
models necessitate manual length control of the generated motions and offer
limited flexibility. In addition to diffusion models, which employ continuous
motion representation, discrete token-based methods utilizing Vector Quantized
Variational Autoencoders (VQ-VAEs) have also demonstrated promising results.
Notable examples include TM2T [10], T2M-GPT [45], MotionGPT [16] and MoMask
[8]. However, most existing works in both approaches encode texts as
conditions for generation, lacking the ability for more comprehensive and
sophisticated language understanding. Moreover, their specialized
architectures can make it challenging when adapting to other tasks within a
generalized framework. By leveraging tokenization methods that encode motions
into a sequence of tokens, it becomes natural to consider harnessing the power
of LLMs that possess strong linguistic capabilities and generalizability for
motion generation.

Motion Captioning Motion Captioning is a significant motion-language task. To
describe human motion using languages, TM2T [10] encodes motions into
sequences of discrete motion tokens, using a translation network to bridge the
gap between motion tokens and text tokens. On the other hand, MotionGPT [16]
pre-trains a motion-language model using language modeling. However, existing
approaches often struggle to produce precise and diverse captions. In our
work, leveraging the linguistic advantages of pre-trained Large Language
Models (LLMs), our MotionLLM achieves state-of-the-art performance in 3D human
motion captioning, generating detailed and accurate text descriptions.

##  3 Method

Figure 1: MotionLLM pipeline. In our multimodal approach, discrete text and
motion tokens work alike in tandem, harnessing LLM in an autoregressive manner
to generate motions or captions. The same motion tokenizing and detokenizing
(namely, encoding and decoding) technique can uniformly handle single or
multiple motions as shown.

As shown in Fig 1, our model consists of three main components: tokenizers, an
LLM for autoregressive generation of motion and text tokens, and detokenizers
for generating the final output. The text tokenizer is inherited from the LLM
and remains unchanged. The motion tokenizer and detokenizer are trained
together to ensure proper reconstruction. Once the tokenizer and detokenizer
are trained, they are kept fixed during the LLM fine-tuning process.

###  3.1 Motion Tokenization

In order to fit better with LLM’s next-token prediction mechanism, we tokenize
motions into discrete representations using Vector Quantization (VQ) and
Variation Encoders (VAE). While [10, 45, 16] utilize VQ-VAEs that contain a
single codebook, [8] uses RVQ-VAE which consists of $Q+1$ ordered codebooks,
where the first one is for the base tokens, similar to traditional VQ, while
the rest are used for residuals and help further improve fidelity. Both
approaches are essentially different variations of quantization, with the
former being a special case that contains only base tokens. In our
experiments, we note that the performance of VQ-VAE and RVQ-VAE is similar for
single-person motion generation, while RVQ-VAE performs better for multi-
person motion generation. We believe this is due to the higher complexity of
interactive motion in multi-person scenarios, which RVQ-VAE can better
capture, including the subtle non-verbal differences in motions.

In our motion tokenization, a motion sequence is represented as
$\textbf{m}_{1:T}\in\mathbb{R}^{T\times D}$ and is first encoded using an
encoder $E$ to motion embeddings $\textbf{z}_{1:T/N}\in\mathbb{R}^{T/N\times
d}$, where $N$ is the downsampling rate and $d$ is the number of the hidden
dimensions. Then the motion embeddings are quantized by a quantizer using
$Q+1$ ($Q=0$ for single codebook) codebooks
$[\textbf{C}^{(q)}=\\{\textbf{c}_{k}^{(q)}\\}_{1}^{K}]_{q=0}^{Q}$, where $K$
is the codebook size and each $\textbf{c}_{k}^{(q)}\in\mathbb{R}^{d}$. For
each embedding $\textbf{z}_{t}$ in the sequence, the closest element in
$\textbf{C}^{(0)}$ is chosen, then their difference is further quantized using
the rest of the codebooks $[\textbf{C}^{(q)}]_{q=1}^{Q}$ layer by layer in
order. The quantization results can be represented as
$[\hat{\textbf{z}}_{1:T/N}^{(q)}]_{q=0}^{Q}$, where

$\hat{\textbf{z}}_{t}^{(0)}=\operatorname*{arg\,min}_{\textbf{c}_{k}\in\textbf{C}^{(0)}}||\textbf{z}_{t}-\textbf{c}_{k}||_{2}$

and for $q=1:Q$, denotes
$\textbf{z}_{t}^{(q)}=\textbf{z}_{t}^{(q-1)}-\hat{\textbf{z}}_{t}^{(q-1)}$
(let $\textbf{z}_{t}^{(0)}=\textbf{z}_{t}$), then,

$\hat{\textbf{z}}_{t}^{(q)}=\operatorname*{arg\,min}_{\textbf{c}_{k}\in\textbf{C}^{(q)}}||\textbf{z}_{t}^{(q)}-\textbf{c}_{k}||_{2}$

The final latent sequence can then be represented as
$\hat{\textbf{z}}_{1:T/N}=\sum_{q=0}^{Q}\hat{\textbf{z}}_{1:T/N}^{(q)}$, and
the original sequence can be reconstructed by the decoder $D$:
$\hat{\textbf{m}}_{1:T}=D(\hat{\textbf{z}}_{1:T/N})$.

We follow [8] to optimize the RVQ-VAE, using reconstruction loss together with
a commitment loss at each quantization layer. We also add an additional
regularization on the joint positions to enhance the generation performance.
The loss can be formulated as:

$\mathcal{L}_{rvq}=\underbrace{||\textbf{m}-\hat{\textbf{m}}||_{1}}_{\mathcal{L}_{re}}+\alpha\underbrace{||\textbf{p}-\hat{\textbf{p}}||_{1}}_{\mathcal{L}_{p}}+\beta\underbrace{\sum_{q=0}^{Q}||\textbf{z}^{(q)}-sg[\hat{\textbf{z}}^{(q)}]||_{2}^{2}}_{\mathcal{L}_{\it{commit}}}$

where $sg[\cdot]$ is the stop-gradient operation, $\alpha$ and $\beta$ are
weighting factors. The codebooks are trained using exponential moving average
(EMA) and codebooks reset following MoMask [8] and T2M-GPT [45]. When the
tokenizers are trained, they are frozen for further usage.

###  3.2 Muitimodal Motion-Language Learning

Following tokenization, the motion representation is discretized into $K$
distinct motion tokens. We utilize the indices of these motion tokens from the
codebook to construct the motion token vocabulary
$\textbf{V}_{m}=\\{{\tt<Motion\\_i>}\\}_{i=1}^{K}$. In addition, we introduce
special tokens "<Motion>" and "</Motion>" to denote the start and end of a
motion token sequence. For multi-human motion generation, we regard this as an
extension to the single-human case and further include a "<Divide>" token to
separate the motions from different people, allowing each person’s motion to
be encoded and decoded separately. These special tokens, together with the
motion tokens, form a new vocabulary set $\textbf{V}_{M}$ of size $K+2$ ($K+3$
for multi-human). This vocabulary is appended to the pre-trained LLM’s
vocabulary.

After expanding the LLM’s vocabulary, a motion can now be denoted as a token
sequence that is understandable by the LLM. During the generation process, the
LLM predicts the succeeding token by maximizing the probability
$p_{\theta}(x_{t}|x_{<t},c)$, where $x_{1:T}$ is the target token sequence and
$c$ represents the prompt. This prediction is performed iteratively in an
autoregressive manner. Consequently, the training objective aims to maximize
the log-likelihood $\mathcal{L}_{\mathit{LLM}}=-\sum\log
p_{\theta}({x_{t}|x_{x_{<t}},c})$.

During the inference process, our approach utilizes instructive prompts such
as “Generate a motion that matches the following input human motion
description.” accompanied by a sentence describing the desired motion. The LLM
then proceeds to predict tokens autoregressively until it predicts the
"</Motion>" token, indicating the completion of the motion generation. This
autoregressive process allows for the generation of motions with variable
lengths, adapting to the specific requirements of the given description.

To fine-tune the LLM, we employ LoRA [13]. Throughout the whole training
process, the tokenizer, the embeddings, and the output layer of the original
text tokens remain unchanged and frozen. Only the additional adapters are
trained. These LoRA adapters are specifically trained for the task at hand
while maintaining a general framework where multiple adapters can coexist
harmoniously. This approach allows us to leverage the power of LLMs while
tailoring them to specific motion-language tasks, ensuring efficient and
effective training without altering the core components of the LLM.

##  4 Experiments

We conduct extensive experiments on various tasks, including single-human
motion generation, multi-human motion generation, and motion captioning. With
both qualitative and quantitative results, we demonstrate that our LLM-based
multimodal language-motion learning framework can be generally applied to
different tasks.

###  4.1 Experiment Setup

Datasets. Our experiments are conducted with KIT Motion Language Dataset (KIT-
ML) [29], HumanML3D [9]. KIT-ML contains 3,911 human motion sequences, while
HumanML3D dataset, obtained from AMASS [23] and HumanAct12 [11], contains
14,616 human motions sequences with 44,970 textual descriptions. As an initial
exploration and extension of our method, we also investigate its performance
in multi-human motion generation using the InterHuman dataset [20], which
provides 7,779 motions with 23,337 textual descriptions.

Single-human motions are represented using ground contact information and
relative joint spatial information with respect to the root. In contrast,
multi-human motions are represented by absolute joint positions and
velocities, allowing for direct tracking of individuals’ trajectories within a
shared world frame. Both representations also incorporate 6D local rotations,
resulting in an over-parameterized structure that is well-suited for neural
network processing. More details are in Sec. A.2.

Evaluation Metric. For motion generation, we follow T2M [9]. Global
representations of motion and textual descriptions are first extracted with
the pre-trained network in [9] and then measured in the following: 1) Text
matching: Conditioned on the motion feature from [9] and 32 text descriptions
(1 ground-truth and 31 randomly selected mismatched descriptions), we
calculate R precision (motion-retrieval precision) by ranking the Euclidean
distances between the motion and text embeddings to obtain Top-1, Top-2, and
Top-3 accuracy of motion-text; additionally for MM Dist (Multimodal distance),
it measures the average Euclidean distances between each text embedding and
the generated motion embeddings from this text; 2) Generation Diversity: we
facilitate the Diversity metric from [9] to measure the variance of the motion
features extracted from generated motions across all descriptions; 3) Motion
similarity, FID (Frechet Inception Distance) measures the distance of motion
features distribution between real and generated motions. For motion
captioning, we follow TM2T [10] to evaluate the quality of motion captioning
by facilitating linguistic metrics from natural language studies, including
Bleu [24], Rouge [21], Cider [37], and Bert Score [48].

Implementation Details. In our tokenizer, we set the downsampling rate $N$ to
4, the hidden dimension $d$ to 512, and the codebook size $K$ to 512. We use a
quantizer containing 1 codebook for all our single-human experiments for
simplicity. In the multi-human case, since it is harder to reconstruct due to
the crucial inter-relative spatial information, we employ a quantizer with 6
codebooks and a detokenizer with more sophistication as detailed in Sec. A.5.
The weighting factors $\alpha$ and $\beta$ for $\mathcal{L}_{p}$ and
$\mathcal{L}_{\it{commit}}$ are set to 0.5 and 0.02 respectively. For the LLM,
we employ Gemma-2b-it [32], a lightweight open-source LLM from Google, which
offers accessibility and can be deployed on a single consumer level GPU. The
LoRA rank is set to 32 for single-human and 64 for multi-human. The values of
alpha remain the same with ranks. All of our experiments are conducted on
NVIDIA RTX4090s.

###  4.2 Results on Single-Human Motion Generation

Quantitative results are shown in Tab. 1, where we compare our model with
state-of-the-art (SOTA) approaches, including diffusion models [34, 4, 46] and
token-based models [45, 16, 8]. Despite fine-tuning only a small number of
parameters, our model performs competitively against these models trained from
scratch. This demonstrates our advantages of leveraging the generalization and
robustness capabilities of LLMs. Additionally, our model exhibits low MMDist
and high Diversity, indicating strong motion-language understanding and
generative capabilities. Note that MoMask [8] and the diffusion model require
known target lengths for generation, and evaluate using ground truth lengths.
However, since the FID metric measures the distance between the distribution
of generated results and ground truth, variable length generated by
autoregressive models can lead to higher FID scores. Our method shows
relatively better performance on the smaller dataset KIT-ML [29] compared to
the state-of-the-art approaches. This could be attributed to the fact that our
model has fewer trainable parameters. In Sec. 4.5, we provide further analysis
and evidence that increasing the model size can lead to overall improvements
in performance scores.

Qualitative Comparisons. In Figure 2, we provide qualitative comparisons
between our approach and three other models: T2M-GPT [45], MotionGPT [16] and
MoMask [8]. We observe that T2M-GPT and MotionGPT can comprehend the “kneel
down” motion but they generate motions where the person kneels down on both
knees. MoMask initially performs the kneeling down motion on one knee but
later transitions back to a double-knees-on-the-floor position, accompanied by
unsynchronized hand motions. All these 3 models fail to understand the concept
of “propose marriage” accurately. As commonly known, proposing marriage
typically involves kneeling down on one knee, which is precisely what our
model correctly presents in the comparison.

Also, the autoregressive nature of our model shows advantages over non-
autoregressive models if they are not given the correct lengths. One example
is presented in Sec. A.1. Furthermore, as an interesting addition, our model
has the ability to process text inputs in languages other than English, even
without prior exposure during our training. For instance, when provided with a
motion description such as “a person first turns left and walks forward”
translated into German, Chinese, and French using an online translator, our
model can still generate the appropriate motion (See Supplementary).

Overall, these results demonstrate the model’s ability to handle highly
customized text input, thereby enhancing its user-friendliness. More
qualitative examples are included in Sec. A.1.

Figure 2: Qualitative comparisons using an unseen text description. Our
MotionLLM is the only model that can generate semantically correct motion.

Datasets | Methods | R Precision $\uparrow$ | FID $\downarrow$ | MultiModal Dist $\downarrow$ | Diversity$\uparrow$
---|---|---|---|---|---
Top 1 | Top 2 | Top 3
| Real | $0.511^{\pm.003}$ | $0.703^{\pm.003}$ | $0.797^{\pm.002}$ | $0.002^{\pm.000}$ | $2.974^{\pm.008}$ | $9.503^{\pm.065}$
T2M [9] | $0.457^{\pm.002}$ | $0.639^{\pm.003}$ | $0.740^{\pm.003}$ | $1.067^{\pm.002}$ | $3.340^{\pm.008}$ | $9.188^{\pm.002}$
TM2T [10] | $0.424^{\pm.003}$ | $0.618^{\pm.003}$ | $0.729^{\pm.002}$ | $1.501^{\pm.017}$ | $3.467^{\pm.011}$ | $8.589^{\pm.076}$
MDM [34] | $0.320^{\pm.005}$ | $0.498^{\pm.004}$ | $0.611^{\pm.007}$ | $0.544^{\pm.044}$ | $5.566^{\pm.027}$ | $9.559^{\pm.086}$
Human |  MLD [4] | $0.481^{\pm.003}$ | $0.673^{\pm.003}$ | $0.772^{\pm.002}$ | $0.473^{\pm.013}$ | $3.196^{\pm.010}$ | $9.724^{\pm.082}$
ML3D |  MotionDiffuse [46] | $0.491^{\pm.001}$ | $0.681^{\pm.001}$ | $0.782^{\pm.001}$ | $0.630^{\pm.001}$ | $3.113^{\pm.001}$ | $9.410^{\pm.049}$
| T2M-GPT [45] | $0.491^{\pm.003}$ | $0.680^{\pm.003}$ | $0.775^{\pm.002}$ | $0.116^{\pm.004}$ | $3.118^{\pm.011}$ | $9.761^{\pm.081}$
| MotionGPT [16] | $0.492^{\pm.003}$ | $0.681^{\pm.003}$ | $0.778^{\pm.002}$ | $0.232^{\pm.008}$ | $3.096^{\pm.008}$ | $9.528^{\pm.071}$
|  MoMask [8] | $\textbf{0.521}^{\pm.002}$ | $\textbf{0.713}^{\pm.002}$ | $\textbf{0.807}^{\pm.002}$ | $\textbf{0.045}^{\pm.002}$ | $\textbf{2.958}^{\pm.008}$ | ${9.620}^{\pm.064}$
| Ours | $0.482^{\pm.004}$ | $0.672^{\pm.003}$ | $0.770^{\pm.002}$ | $0.491^{\pm.019}$ | $3.138^{\pm.010}$ | $\textbf{9.838}^{\pm.244}$
| Real | $0.424^{\pm.005}$ | $0.649^{\pm.006}$ | $0.779^{\pm.006}$ | $0.031^{\pm.004}$ | $2.788^{\pm.012}$ | $11.080^{\pm.097}$
T2M [9] | $0.361^{\pm.006}$ | $0.559^{\pm.007}$ | $0.681^{\pm.007}$ | $3.022^{\pm.107}$ | $3.488^{\pm.028}$ | $10.720^{\pm.145}$
TM2T [10] | $0.280^{\pm.006}$ | $0.463^{\pm.007}$ | $0.587^{\pm.005}$ | $3.599^{\pm.153}$ | $4.591^{\pm.026}$ | $9.473^{\pm.117}$
MDM [34] | $0.164^{\pm.004}$ | $0.291^{\pm.004}$ | $0.396^{\pm.004}$ | $0.497^{\pm.021}$ | $9.191^{\pm.022}$ | $10.847^{\pm.109}$
KIT- |  MLD [4] | $0.390^{\pm.008}$ | $0.609^{\pm.008}$ | $0.734^{\pm.007}$ | $0.404^{\pm.027}$ | $3.204^{\pm.027}$ | $10.800^{\pm.117}$
ML |  MotionDiffuse [46] | $0.417^{\pm.004}$ | $0.621^{\pm.004}$ | $0.739^{\pm.004}$ | $1.954^{\pm.062}$ | $2.958^{\pm.005}$ | $11.100^{\pm.143}$
| T2M-GPT [45] | $0.416^{\pm.006}$ | $0.627^{\pm.006}$ | $0.745^{\pm.006}$ | $0.514^{\pm.029}$ | $3.007^{\pm.023}$ | $10.921^{\pm.108}$
| MotionGPT [16] | $0.366^{\pm.005}$ | $0.558^{\pm.004}$ | $0.680^{\pm.005}$ | $0.510^{\pm.016}$ | $3.527^{\pm.021}$ | $10.350^{\pm.084}$
|  MoMask [8] | $\textbf{0.433}^{\pm.007}$ | $\textbf{0.656}^{\pm.005}$ | $\textbf{0.781}^{\pm.005}$ | $\textbf{0.204}^{\pm.011}$ | $\textbf{2.779}^{\pm.022}$ | $10.711^{\pm.087}$
| Ours | $0.409^{\pm.006}$ | $0.624^{\pm.007}$ | $0.750^{\pm.005}$ | $0.781^{\pm.026}$ | $2.982^{\pm.022}$ | $\textbf{11.407}^{\pm.103}$

Table 1: Quantitative evaluation on the HumanML3D [9] and KIT-ML [29] test
set. We follow T2M [9] for the evaluation metrics. The evaluations are
conducted 20 times to obtain a 95% confidence interval. Methods indicated in
italics utilize the ground truth lengths for estimation. Bold indicates the
best result.

###  4.3 Results on Motion Captioning

As our motion-aware LLM model is token-based, it possesses the ability to
generate text tokens from a given human motion sequence. We compare our motion
captioning results with current SOTA models in handling motion-to-text tasks,
using a hybrid evaluation method that combines quantitative analysis,
qualitative assessment, and user study approaches.

Inheriting the strong text processing capabilities of LLMs, MotionLLM can
generate faithful descriptions of human motions.

#### Quantitative results

We evaluate the generated captions based on the linguistic metrics suggested
in [10], where semantic similarities with ground truth captions are calculated
to examine the accuracy of the generated captions. Since[10] pre-processes the
text by ignoring grammatical tense and plural forms of words, we follow [16]
to use ground truth texts for a more accurate assessment.

As shown in Tab. 2, our method achieves state-of-the-art performance across
all metrics, significantly outperforming both MotionGPT [16] and TM2T [10].

Methods | Bleu@1$\uparrow$ | Bleu@4$\uparrow$ | Rouge$\uparrow$ | Cider$\uparrow$ | Bert Score$\uparrow$
---|---|---|---|---|---
TM2T [10] | 48.90 | 8.27 | 38.1 | 16.80 | 32.2
MotionGPT [16] | 48.20 | 12.47 | 37.4 | 29.20 | 32.4
Ours | 54.26 | 16.66 | 41.2 | 42.34 | 39.8
Table 2: Comparison on motion-to-text captioning on HumanML3D [9] test set. We
use the ground truth captions without pre-processing and linguistic metrics
suggested in [10] for evaluation. Best scores are highlighted in bold.

Qualitative Comparisons. Notably, our model has a clear advantage over other
models in its ability to capture the true characteristics of human motion. For
example, MotionLLM can generate descriptions such as “walking like a zombie”
rather than simply describing the motions as “walking with their arms in front
of their chest.”

For less obvious and subtle motions with unclear patterns or clear intention,
MotionLLM generates detailed descriptions that include information about body
parts (“left hand”), actions (“is waving”), and spatial information (“above
the head”). This detailed approach ensures a more accurate and nuanced
depiction of human motion. More qualitative results are in Sec. A.1.

User Study. Furthermore, we conducted a crowdsourced user assessment to
evaluate the quality of our captioning results. Specifically, we randomly
sampled 17 motions from HumanML3D test set and carried out two sets of tests.
Each test involved a combination of the ground truth caption and the caption
generated by our model, MotionLLM, as well as an alternative caption from
other models, namely TM2T or MotionGPT. The alternative model remains
consistent among a set of tests.

Participants were asked to rank three captions for each motion based on their
accuracy in describing the given motions, from most to least accurate.

We received high-quality feedback from a diverse range of social groups, with
a total of 25 responses for the MotionGPT set and 25 responses for the TM2T
set. As illustrated in Fig. 3, our method consistently produces captions that
are rated as more accurate compared to those generated by other models, and
are even competitive with the ground truth captions.

Figure 3: User study where participants were asked to assign one of the three
accuracy ratings (above the charts) to the given captions generated by
different models (below).

###  4.4 Extending to Multi-Human Motion Generation

MotionLLM can naturally handle multi-human motion generation, and the results
demonstrate significant potential and generalizability in this task. Figure 4
shows qualitative results on InterHuman [20] test set. Despite some good
results, we observe that multi-human motion generation is considerably more
complex than single-human motions, with more ambiguous text descriptions and
correlated motions among interacting humans. We believe more data should be
required compared to the single-human case to train a large-scale model.

Given the ongoing challenges on multi-human generation, in this preliminary
study, as we have introduced in Sec.3.2, we use a "<Divide>" token to separate
the two interactive humans and generate in one single inference. Additionally,
we experimented an iterative inference approach without "<Divide>", e.g.,
first prompt “Generate the first person’s motion of two interactive humans”
and then use the generated motion token sequence as a condition to generate
the second motion: “Generate the second person’s motion of two interactive
humans given the first person’s motion.” We observe that the results are
similar, which is expected since they are essentially the same. Due to LLM’s
autoregressive nature, in the method of using "<Divide>", the second person’s
motion is also conditioned on the first one. Though producing similar results,
the method using "<Divide>" offers better scalability, reduces inference time,
and has the potential to be extended to include more than two individuals by
simply incorporating additional "<Divide>" tokens.

Figure 4: Results on multi-human motion generation.

###  4.5 Ablation Study

We conducted an ablation study to examine the impact of different LLM
backbones and adapter sizes. The results are shown in Tab. 3, from which we
may conclude that using larger backbone models or increasing the LoRA rank
leads to overall improvements in the metrics. However, it is noteworthy that
fine-tuning all the parameters of the LLM does not yield significant
improvements compared to the increase in trainable parameters. While the FID
metric shows relatively more improvement due to the model’s ability to learn
the data distribution with more parameters, the generalizability decreases, as
indicated by the lower diversity score. This implies that the model becomes
more rigid and less capable of generating diverse motions when trained on all
parameters. In contrast, our adapter-based training approach demonstrates high
efficiency, with its plug-and-play feature enabling greater flexibility and
portability as well.

Also, it is interesting to notice that, using Gemma-2b which is a not-
instruction-tuned version of Gemma [32], the performance is lower compared to
Gemma-2b-it. We attribute this to the limited instruction and prompt
understanding ability of the base model, and conjecture that using an
instruction-tuned LLM backbone may give better results in multimodal learning.

Models | Trainable Params | R Precision $\uparrow$ | FID $\downarrow$ | Multimodal Dist $\downarrow$ | Diversity $\uparrow$
---|---|---|---|---|---
Top 1 | Top 3
Real | - | 0.424 | 0.779 | 0.031 | 2.788 | 11.08
Gemma-2b-it R=16 | 19.6M | 0.386 | 0.711 | 1.089 | 3.210 | 11.132
Gemma-2b R=32 | 39.2M | 0.388 | 0.720 | 0.993 | 3.189 | 11.002
Gemma-2b-it R=32 | 39.2M | 0.409 | 0.750 | 0.781 | 2.982 | 11.407
Gemma-2b-it R=64 | 78.4M | 0.410 | 0.751 | 0.725 | 2.966 | 11.198
LLaMA-3-8B-Instruct R=32 | 83.9M | 0.381 | 0.737 | 0.646 | 3.046 | 11.210
Gemma-7b-it R=32 | 100.0M | 0.429 | 0.747 | 0.515 | 2.800 | 11.165
Gemma-2b-it Full | 2.5B | 0.411 | 0.723 | 0.525 | 2.824 | 10.685

Table 3: Results of ablation study on KIT-ML [29]. Gemma [32] and LLaMA [35]
are chosen as LLM backbones. R indicates the LoRA rank, the value of alpha is
kept the same with the rank. "Full" indicates that the model is fully fine-
tuned.

##  5 Discussion

#### Limitations.

Though capable of generating high-quality results, our model suffers from a
relatively long inference time. Also, though we are the first to use discrete
tokens for multi-human generation and can achieve some promising results, the
overall performance in multi-human motion generation still somewhat falls
short compared to our single-human results. This can be attributed to several
challenges, including the limited availability of high-quality data and the
increased complexity and ambiguity inherent in multi-human motion language
descriptions. Nonetheless, our methods exhibit strong potential and hold
promise for further improvements in multi-human cases. We will leave the
improvement in our future work.

#### Concluding Remarks.

In this work, we propose a novel LLM-based motion-language learning framework,
offering scalability and flexibility. By harnessing the linguistic
comprehension and generation capabilities of LLMs, our approach achieves good
results in bidirectional translation between motion and natural language.
Furthermore, without training a large model from scratch, we showcase how our
straightforward and adaptable framework holds promising potential for various
tasks, including multi-human motion generation. Our current results are
generated using Gemma-2b-it, one of the smallest LLMs released to the public.
We believe better results could be obtained with larger and more advanced LLM
models.

## References

  * [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [2] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour.  Audiolm: a language modeling approach to audio generation.  arXiv preprint arXiv:2209.03143, 2023.
  * [3] Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu.  X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages.  arXiv preprint arXiv:2305.04160, 2023.
  * [4] Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu.  Executing your commands via motion diffusion in latent space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18000–18010, 2023.
  * [5] Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, and Christian Theobalt.  Mofusion: A framework for denoising-diffusion-based motion synthesis.  In Computer Vision and Pattern Recognition (CVPR), 2023.
  * [6] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao.  Llama-adapter v2: Parameter-efficient visual instruction model.  arXiv preprint arXiv:2304.15010, 2023.
  * [7] Purvi Goel, Kuan-Chieh Wang, C. Karen Liu, and Kayvon Fatahalian.  Iterative motion editing with natural language.  arXiv preprint arXiv:2312.11538, 2023.
  * [8] Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng.  Momask: Generative masked modeling of 3d human motions.  arXiv preprint arXiv:2312.00063, 2023.
  * [9] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng.  Generating diverse and natural 3d human motions from text.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152–5161, June 2022.
  * [10] Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng.  Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts.  In European Conference on Computer Vision, pages 580–597. Springer, 2022.
  * [11] Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng.  Action2motion: Conditioned generation of 3d human motions.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 2021–2029, 2020.
  * [12] Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue.  Onellm: One framework to align all modalities with language.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
  * [13] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [14] Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Zhou Zhao, and Shinji Watanabe.  Audiogpt: Understanding and generating speech, music, sound, and talking head.  arXiv preprint arXiv:2304.12995, 2023.
  * [15] Yiming Huang, Weilin Wan, Yue Yang, Chris Callison-Burch, Mark Yatskar, and Lingjie Liu.  Como: Controllable motion generation through language guided pose code editing.  arXiv preprint arXiv:2403.13900, 2024.
  * [16] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen.  Motiongpt: Human motion as a foreign language.  Advances in Neural Information Processing Systems, 36, 2024.
  * [17] Biao Jiang, Xin Chen, Chi Zhang, Fukun Yin, Zhuoyuan Li, Gang YU, and Jiayuan Fan.  Motionchain: Conversational motion controllers via multimodal prompts.  arXiv preprint arXiv:2404.01700, 2024.
  * [18] Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov.  Generating images with multimodal language models.  In Advances in Neural Information Processing Systems, volume 36, 2024.
  * [19] Taeryung Lee, Gyeongsik Moon, and Kyoung Mu Lee.  Multiact: Long-term 3d human motion generation from multiple action labels.  In AAAI Conference on Artificial Intelligence (AAAI), 2023.
  * [20] Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu.  Intergen: Diffusion-based multi-human motion generation under complex interactions.  International Journal of Computer Vision, pages 1–21, 2024.
  * [21] Chin-Yew Lin.  ROUGE: A package for automatic evaluation of summaries.  In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
  * [22] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel.  World model on million-length video and language with ringattention.  arXiv preprint arXiv:2402.08268, 2024.
  * [23] Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black.  AMASS: Archive of motion capture as surface shapes.  In International Conference on Computer Vision, pages 5442–5451, October 2019.
  * [24] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  * [25] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3d hands, face, and body from a single image.  In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [26] Mathis Petrovich, Michael J. Black, and Gül Varol.  Action-conditioned 3D human motion synthesis with transformer VAE.  In International Conference on Computer Vision (ICCV), 2021.
  * [27] Mathis Petrovich, Michael J. Black, and Gül Varol.  TEMOS: Generating diverse human motions from textual descriptions.  In European Conference on Computer Vision (ECCV), 2022.
  * [28] Mathis Petrovich, Or Litany, Umar Iqbal, Michael J. Black, Gül Varol, Xue Bin Peng, and Davis Rempe.  Multi-track timeline control for text-driven 3d human motion generation.  In CVPR Workshop on Human Motion Generation, 2024.
  * [29] Matthias Plappert, Christian Mandery, and Tamim Asfour.  The KIT motion-language dataset.  Big Data, 4(4):236–252, dec 2016.
  * [30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020, 2021.
  * [31] Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H. Bermano.  Priormdm: Human motion diffusion as a generative prior.  In International Conference on Learning Representations, 2024.
  * [32] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al.  Gemma: Open models based on gemini research and technology.  arXiv preprint arXiv:2403.08295, 2024.
  * [33] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or.  Motionclip: Exposing human motion generation to clip space.  arXiv preprint arXiv:2203.08063, 2022.
  * [34] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H. Bermano.  Mdm: Human motion diffusion model.  In International Conference on Learning Representations, 2023.
  * [35] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [36] Jonathan Tseng, Rodrigo Castellon, and C Karen Liu.  Edge: Editable dance generation from music.  arXiv preprint arXiv:2211.10658, 2022.
  * [37] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  arXiv preprint arXiv:1411.5726, 2015.
  * [38] Weilin Wan, Zhiyang Dou, Taku Komura, Wenping Wang, Dinesh Jayaraman, and Lingjie Liu.  Tlcontrol: Trajectory and language control for human motion synthesis.  arXiv preprint arXiv:2311.17135, 2023.
  * [39] Yin Wang, Zhiying Leng, Frederick W. B. Li, Shun-Cheng Wu, and Xiaohui Liang.  Fg-t2m: Fine-grained text-driven human motion generation via diffusion model.  arXiv preprint arXiv:2309.06284, 2023.
  * [40] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  Next-gpt: Any-to-any multimodal llm.  arXiv preprint arXiv:2309.05519, 2023.
  * [41] Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang.  Omnicontrol: Control any joint at any time for human motion generation.  In ICLR, 2024.
  * [42] Liang Xu, Ziyang Song, Dongliang Wang, Jing Su, Zhicheng Fang, Chenjing Ding, Weihao Gan, Yichao Yan, Xin Jin, Xiaokang Yang, et al.  Actformer: A gan-based transformer towards general action-conditioned 3d human motion generation.  ICCV, 2023.
  * [43] Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin.  Pointllm: Empowering large language models to understand point clouds.  arXiv preprint arXiv:2308.16911, 2023.
  * [44] Hang Zhang, Xin Li, and Lidong Bing.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  arXiv preprint arXiv:2306.02858, 2023.
  * [45] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen.  T2m-gpt: Generating human motion from textual descriptions with discrete representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [46] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu.  Motiondiffuse: Text-driven human motion generation with diffusion model.  arXiv preprint arXiv:2208.15001, 2022.
  * [47] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu.  Motiondiffuse: Text-driven human motion generation with diffusion model.  arXiv preprint arXiv:2208.15001, 2022.
  * [48] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  arXiv preprint arXiv:1904.09675, 2020.
  * [49] Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang.  Motiongpt: Finetuned llms are general-purpose motion generators.  arXiv preprint arXiv:2306.10900, 2023.
  * [50] Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei.  Speak foreign languages with your own voice: Cross-lingual neural codec language modeling.  arXiv preprint arXiv:2303.03926, 2023.
  * [51] Wenyang Zhou, Zhiyang Dou, Zeyu Cao, Zhouyingcheng Liao, Jingbo Wang, Wenjia Wang, Yuan Liu, Taku Komura, Wenping Wang, and Lingjie Liu.  Emdm: Efficient motion diffusion model for fast, high-quality motion generation.  arXiv preprint arXiv:2312.02256, 2023.
  * [52] Zixiang Zhou and Baoyuan Wang.  Ude: A unified driving engine for human motion generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5632–5641, 2023.

##  Appendix A Appendix

In the appendix, we present:

  * •

Section A.1: More Qualitative Results.

  * •

Section A.2: Data representations of single-human, multi-human motions.

  * •

Section A.3: Details of our user study.

  * •

Section A.4: More details about the evaluation metrics.

  * •

Section A.5: More details regarding our implementation, including an ablation
study on our choice of VQ and RVQ.

###  A.1 Qualitative Results

Rendered original videos of all examples shown in the paper can be found in
the corresponding folder of the supplementary material.

#### Motion Generation

Comparison on no-length-given motion generation. More qualitative results are
in the supplementary material.

Figure 5: Comparison between ours and MoMask [8], which is non-autoregressive.
During regular inference, MoMask uses a length estimator to predict the length
conditioned on the text. This estimator is likely to fail. In this example,
their incorrect predicted length causes severe drifting.

#### Motion Captioning

Motion | Model | Caption
---|---|---
| Ground Truth | a person walks forward just like a mummy
| TM2T | a person walk in a counterclockwise circle with their arm out to the side
| MotionGPT | the person is walking like a mummy from the dead
demo_1 | Ours | a person walks forward while holding arms out as if to be a zombie
| Ground Truth | a person walks forward slowly while their right hand is slightly elevate
| TM2T | a person slowly walk forward while hold onto something with their left hand
| MotionGPT |  | a person walks forward slowly, placing one foot in front of the other, on a belt
---
that circulates, enabling the person to effectively slowly walk in place.
demo_2 | Ours | the person is walking on a balance beam
| Ground Truth | a person moves side to side in a zigzag fashion backwards
| TM2T | a person does a cartwheel to the right
| MotionGPT | a person is practing defense moves.
demo_3 | Ours | a person walks backwards in zig-zag motion
| Ground Truth | a person makes and drinks a cup of coffee
| TM2T |  | person hold something with their right hand and make a sawing motion with
---
their left hand
| MotionGPT | a person is eating something
demo_4 | Ours |  | a person uses their left hand to open a bottle, drinks from it, then places the
---
bottle back down

Table 4: Comparsion of motion captioning ability across different models.
Original motions can be found in supplementary material.

Our model is capable of generating high-quality motion captions,
demonstrations are provided in Table 4.

###  A.2 Data Representation

For single-human motion, we follow [9], a 3D human motion sequence $X$ in our
work is represented canonically as

$x^{i}=[\dot{r}^{a},\dot{r}^{x},\dot{r}^{z},r^{y},j^{p},j^{v},j^{r},c^{f}]$

where $x^{i}$, the i-th motion is the combinations of root angular velocity
along Y-axis $(\dot{r}^{a}\in\mathbb{R}$), root linear velocities on XZ-plane
$(\dot{r}^{x},\dot{r}^{z}\in\mathbb{R})$, root height $(r^{y}\in\mathbb{R}$),
local joints positions, velocities and rotations in root
space$(j^{p}\in\mathbb{R}^{3j}$, $j^{v}\in\mathbb{R}^{3j}$ and
$j^{r}\in\mathbb{R}^{6j}$), binary features obtained by thresholding the heel
and toe joint velocities $(c^{f}\in\mathbb{R}^{4}$). $j$ denotes the number of
joints. Motions in the HumanML3D dataset follow the skeleton structure of SMPL
[25] with 22 joints, while motions have 21 joints in KIT-ML dataset.

For multi-human motions, we follow [20], one human in the motion sequence $X$
is represented non-canonically as

$x^{i}=[j^{p},j^{v},j^{r},c^{f}],$

where $j^{p}\ \in\mathbb{R}^{3j}$ represents global joint positions, $j^{v}\
\in\mathbb{R}^{3j}$ represents velocities, $j^{p}\ \in\mathbb{R}^{6j}$
represents 6D rotations in root space, and $c^{f}\in\mathbb{R}^{4}$ represents
binary foot-ground contact features. $j$ denotes the number of joints for each
persion. Specifically, motions in InterHuman dataset has 22 joints for each
person.

###  A.3 User Study

Figure 6: Screenshot of the crowd-sourced survey.

  * •

Description: Participants were informed that participation in the survey is
entirely voluntary. The survey introduction explicitly stated that they could
withdraw at any time without any consequences.

  * •

Details: The survey included information about the purpose of the study,
estimated time commitment, confidentiality assurances, and contact information
for any questions or concerns.

###  A.4 Evaluation Metric

We detail the calculation of several evaluation metrics proposed in [9]. We
denote ground-truth motion features, generated motion features, and text
features as $f_{\text{gt}}$, $f_{\text{pred}}$, and $f_{\text{text}}$. Note
that these features are extracted with pretrained networks in [9].

Multimodal Distance (MM-Dist). MM-Dist is widely used to evaluate the motion
generation ability of the model. MM-Dist measures the distance between the
text embedding and the generated motion feature. Given $N$ randomly generated
samples, the MM-Dist measures the feature-level distance between the motion
and the text. It computes the average Euclidean distances between each text
feature and the generated motion feature from this text:

$\text{MM-
Dist}=\frac{1}{N}\sum_{i=1}^{N}\|f_{\text{pred},i}-f_{\text{text},i}\|$

where $f_{\text{pred},i}$ and $f_{\text{text},i}$ are the features of the
$i$-th text-motion pair.

Frechet Inception Distance (FID). FID measures the distance of motion features
distribution between real and generated motions. We calculate FID by

$\mathit{FID}=\|\mu_{\text{gt}}-\mu_{\text{pred}}\|^{2}-\text{Tr}(\Sigma_{\text{gt}}+\Sigma_{\text{pred}}-2(\Sigma_{\text{gt}}\Sigma_{\text{pred}})^{1/2})$

where $\mu_{\text{gt}}$ and $\mu_{\text{pred}}$ are the means of
$f_{\text{gt}}$ and $f_{\text{pred}}$. $\Sigma$ is the covariance matrix and
$\operatorname{Tr}$ denotes the trace of a matrix.

R precision Given the motion sequence and 32 text descriptions (1 ground-truth
and 31 randomly selected mismatched descriptions), we rank the Euclidean
distances between the motion and text embeddings to get Top-1, Top-2, and
Top-3 accuracy of motion-text;

Diversity. Diversity measures the variance of the whole motion sequences
across the dataset. We randomly sample $S_{\text{dis}}$ pairs of motion and
each pair of motion features is denoted by $f_{\text{pred},i}$ and
$f^{\prime}_{\text{pred},i}$. The diversity can be calculated by

$\mathit{Diversity}=\frac{1}{S_{\text{dis}}}\sum_{i=1}^{S_{\text{dis}}}\|f_{\text{pred},i}-f^{\prime}_{\text{pred},i}\|$

In our experiments, we set $S_{\text{dis}}$ to 300 as [9].

Linguistic metrics. Linguistic metrics including Bleu [24], Rouge [21], Cider
[37] and Bert Score [48], we follow TM2T [10], using NLPEval to calculate.
Readers can refer to their papers for further details.

###  A.5 More Implementation details

#### RVQ-VAE.

In multi-human motion generation, we find that it’s hard for original VQ-VAEs
to reconstruct the motions. Since multi-human motions are represented using
absolute joint locations, the loss of information caused by quantization may
result in problems such as drifting. Hence we use RVQ for less quantization
loss, following [8]. In this case, since the LLM is only aware of the base
layer tokens, an additional non-autoregressive transformer is needed to
predict the $Q$ (in our case $Q=5$) residual layer token sequences given the
base layer token from the LLM output. This non-autoregressive transformer can
be regarded as a part of our detokenizer (or namely decoder), and has no other
influence on the LLM inference/training and our overall generalized framework.
We follow [8] to implement this transformer. This transformer has 8 layers, 6
heads, with $\text{latent\\_dim}=384$, $\text{ff\\_size}=1024$,
$\text{dropout}=0.2$. The non-autoregressive transformer contains $Q$ separate
embedding layers, each for one residual layer token prediction. During
training, one of the $Q$ residual layers is randomly chosen to be learned.

#### Ablation on VQ and RVQ.

As we have discussed in our main paper, we choose VQ instead of RVQ for
single-human motion, this is simply because we find that VQ is already
sufficient for single-human motion generation since single-human motions don’t
need to incorporate absolute spatial information. As in Tab. 5, the
involvement of RVQ and hence the non-autoregressive transformer introduce
about $\nicefrac{{1}}{{3}}$ more trainable parameters, but only result in
limited improvement. Therefore, we choose VQ for the single-human case to
minimize redundancy and training overhead.

Models | Trainable Params | R Precision $\uparrow$ | FID $\downarrow$ | Multimodal Dist $\downarrow$ | Diversity $\uparrow$
---|---|---|---|---|---
Top 1 | Top 3
MotionLLM with RVQ | 39.2M + 13.4M | 0.409 | 0.750 | 0.724 | 2.809 | 10.360
MotionLLM with VQ | 39.2M | 0.409 | 0.750 | 0.781 | 2.982 | 11.407

Table 5: Ablation study of different quantizing strategy on KIT-ML [29]. The
"13.4M" trainable parameters are from the non-autoregressive transformer.

#### Prompts For LLMs.

We use different prompts for different tasks.

Task | Prompts
---|---
Single-human Motion Generation |  Generate a motion matching the following input human motion description.
Motion Captioning |  Generate a caption matching the following input human motion token sequence.
Multi-human Motion Generation |  Generate motions of two interacting humans matching the following input human motions description. Two people’s motion should be divided using <Divide>.
Table 6: Instructing prompts for LLM training and inference

#### Hyper-parameters.

Our hyper-parameters settings for different tasks.

Hyper-parameter | Single-human Motion Generation | Motion Captioning | Multi-human Motion Generation
---|---|---|---
Batch size | 6 | 6 | 6
Learning rate | 1e-5 | 1e-5 | 1e-5
LoRA rank | 32 | 32 | 64
LoRA alpha | 32 | 32 | 64
LoRA dropout | 0.1 | 0.1 | 0.1
Codebook size | 512 | 512 | 512
Codebook dim | 512 | 512 | 512
Total vocab size | 256514 | 256514 | 256515

Table 7: Hyper-parameters of our models used in our main experiments. Other
VQ/RVQ training settings are borrow from T2M-GPT [45] and MoMask [8].


# Fully Convolutional Networks for Handwriting Recognition

Felipe Petroski Such*, Dheeraj Peri*, Frank Brockler†, Paul Hutkowski†,
Raymond Ptucha*
*Rochester Institute of Technology, USA, †Kodak Alaris, USA
{rwpeec, fps7806<EMAIL_ADDRESS>
###### Abstract

Handwritten text recognition is challenging because of the virtually infinite
ways a human can write the same message. Our fully convolutional handwriting
model takes in a handwriting sample of unknown length and outputs an arbitrary
stream of symbols. Our dual stream architecture uses both local and global
context and mitigates the need for heavy preprocessing steps such as symbol
alignment correction as well as complex post processing steps such as
connectionist temporal classification, dictionary matching or language models.
Using over 100 unique symbols, our model is agnostic to Latin-based languages,
and is shown to be quite competitive with state of the art dictionary based
methods on the popular IAM and RIMES datasets. When a dictionary is known, we
further allow a probabilistic character error rate to correct errant word
blocks. Finally, we introduce an attention based mechanism which can
automatically target variants of handwriting, such as slant, stroke width, or
noise.

Keywords\- Fully Convolutional Neural Networks, Handwriting Recognition, Deep
Learning

##  I INTRODUCTION

Convolution Neural Networks (CNN) have enabled many recent successes in
pattern recognition systems. For example, deep CNNs have become ubiquitous in
classification, segmentation and object detection. This success has also been
demonstrated in Optical Character Recognition (OCR) systems, where CNNs can
predict a sequence of characters from machine generated text with near-perfect
accuracy.

Despite a shift towards digitized information exchange, there is still a need
for handwritten inputs in many documents such as invoices, taxes, memos, and
questionnaires. Intelligent Character Recognition (ICR) is the task of
deciphering digitized handwritten text. ICR is quite a bit more challenging
than OCR because no two handwritten symbols are identical. ICR handwriting
systems can be online or offline. The former records stroke sequences say on a
tablet, while the latter has no temporal information. This paper is concerned
with offline ICR.

Like OCR, ICR systems first extract lines of text and optionally can segment
lines into word blocks separated by white space. ICR systems can feed these
word blocks of symbols, often in context with surrounding lines of text, into
lexicon-based classifiers. Constraining the output to a lexicon of words can
result in very accurate systems, but these same systems unfortunately cannot
generalize to commonly occurring sequences such as addresses, phone numbers,
and surnames. Removal of dictionary lookup allows the recognition of unbounded
dictionaries, but at the expense of decreased accuracy.

Jaderberg et al. [16] proposed a framework which does word level recognition
using CNNs on OCR tasks. Poznanski and Wolf [23] used deep CNNs to extract
n-grams which feed Canonical Correlation Analysis (CCA) for final word
recognition from a fixed vocabulary.

A variant of Recurrent Neural Networks (RNNs), named Long Short Term Memory
(LSTM) [14] has shown incredible progress in capturing long term dependencies
and have been used in sequence prediction. Some works [21, 5, 22, 9] split an
image into segments, then feed the sequence of segments into a RNN to predict
a corresponding sequence of output characters. Connectionist Temporal
Classification (CTC) [13] further eliminates the need for precise alignment.
Xie et al. [31] used CNNs to feed a multi-layer LSTM network for handwritten
Chinese character recognition. Similar techniques have also been used for text
recognition in natural imagery [25, 26].

A combination of convolutional networks and recurrent neural networks have
been used by [28, 30] which use convolution layers for feature extraction and
recurrent layers as sequence predictors. [30] performed ICR at the paragraph
level to include language context.

Fully Convolutional neural Network (FCN) methods [19, 7] take in arbitrary
size images and output pixel level classification. Extracted portions of
handwritten text have arbitrary length and can benefit from FCN methods. By
using a CNN to estimate the number of symbols in a word block, word blocks can
be resized to a canonical representation tuned to a FCN architecture. Knowing
the average symbol width, a FCN model can perform accurate symbol prediction
without CTC post processing.

This paper proposes a method to obtain character based classification without
relying on predefined dictionaries or contextual information. We believe our
method is the first that can reliably predict both arbitrary symbols as well
as words from a dictionary using a single architecture without any pre- or
post-processing. The novel contributions of this paper are: 1) Introduction of
a dual stream fully convolutional CNN architecture for accurate symbol
prediction; 2) Usage of a probabilistic character error rate that calculates a
word probability from a sequence of character probabilities; and 3) Creation
of a difficult, but realistic block based dataset derived from the recently
released NIST single character dataset [1].

Figure 1: The Symbol CNN is a fully convolutional model comprised of two
networks. The bottom part of the network takes an input image of size
$32\times 128N$, where $N$ is the number of symbols in the word block. The top
part of the network processes an image of size $32\times 128$ and adds context
during symbol prediction.

##  II Related Work

Recent works in scene text recognition [32, 2] use convolutional neural
networks and pre-processing techniques like adjusting orientations to predict
characters from a scene text. Batuhan et al. [4] proposed an end-to-end
framework of recognizing characters from a document by using LSTMs for
character segmentation and CNNs for character classification.

One of the advantages of using CNNs is that inputs can be unprocessed data
such as raw pixels of an image, rather than extracting specific features [29]
or pen stroke grid values. Connectionist Temporal Classification (CTC) removes
the need to forcefully align the input stream with character prediction
locations [13]. One of the major advantages of the CTC algorithm is that you
do not need properly segmented labeled data. Although the CTC algorithm takes
care of the alignment of input with the output labels, it can be finicky to
train and increases compute complexity during inference.

Huang and Srihari [15] described an approach to separate a line of handwritten
text to words. They proposed a gap metrics based approach to perform word
segmentation task. Rather than segmenting words, Gader et al. [12] proposed
character segmentation utilizing information as you move from background
pixels to foreground pixels in horizontal and vertical directions of the
character image.

Doetsch et al. [9] proposed hybrid RNN-HMM for English offline handwriting
recognition. They introduced a new variant of a LSTM memory cell by using a
scalar multiple for every gate in each layer of the RNN. The scaling technique
for LSTM gates reduced Character Error Rate (CER) by 0.3%. Bluche et al. [1]
compared CNN and traditional feature extraction techniques along with HMM for
transcription. Pham et al. [22] proposed Multidimensional RNN using dropout to
improve offline handwriting recognition accuracy by 3%.

Dewan and Srinivasa [8] and Xie et al. [31] used CNNs for offline character
recognition of Telugu and Chinese characters respectively. [8] used auto
encoders, where the model was trained in a greedy layer wise fashion to learn
weights in an unsupervised fashion, then fine-tuned with supervised data.

##  III Methods

An important part of character recognition is Region of Interest (ROI)
extraction. Areas of text are extracted from documents using regional based
classifiers such as R-CNNs [24] or pixel based segmentation [7]. These regions
are split into lines of text using modified XY tree [6]. Each line of text is
then split into word blocks, where a word block is a string of symbols
separated by white space. The string of symbols could be a word, phone number,
surname, acronym, etc. Word block boundaries can again be determined by
modified XY tree. Using punctuation detectors [27], it is possible to detect
and store a string of word blocks to form sentences, and possible to detect
and store a string of sentences to form paragraphs. The FCNs in this research
take extracted word blocks as input. Preprocessing such as contrast
normalization and deslanting have shown to be effective for handwriting
recognition [18] [30], but are not used in this research.

We propose a three staged approach in this paper for the recognition of
handwritten characters. In the first stage, we train a CNN to quickly predict
the word label for common words such as “the”, “her”, “this”, etc. This model
is a typical image classification model where the ground truth classes
correspond to discrete words from the training set. We call this a Vocabulary
CNN, as it can only predict the common lexicon of words from the training set.
The architecture of the Vocabulary CNN is
C(64,3,3)-C(64,3,3)-C(64,3,3)-P(2)-C(128,3,3)-C(128,3,3)-C(256,3,3)-P(2)-C(256,3,3)-C(512,3,3)-C(512,3,3)-P(2)-C(256,4,16)-FC(V)-SoftMax
where C(D,H,W) stands for convolution with the dimensions of the filter as
$H{\times}W$ and the depth $D$. Each convolutional layer is followed by a
batch norm and ReLU layer. P(2) represents a 2$\times$2 pooling layer with
stride 2. The last layer FC(V) is a fully connected layer with V being the
lexicon size. We consider a prediction to be valid if the confidence level is
more than 0.7, else we process the block through the next two stages.

In the second stage, we train a CNN to predict the number of symbols in a word
block. We refer to this CNN as the Length CNN. This model has a similar
architecture to the Vocabulary CNN with additional max-out layers in between
convolutional layers. The ground truth classes correspond to the length of the
word blocks in the training set. The predicted number of symbols from the
Length CNN is used to resize the word block to a canonical representation of
$32\times 128N$, where $N$ is the number of symbols. This resized word block
is then passed into the third stage which is a fully convolutional variant of
CNN.

In the third stage, a dual stream FCN predicts an arbitrary length sequence of
characters from a variable length word block. FCN’s are a natural choice for
processing variable length word blocks as regular CNNs require fixed length
inputs. We call this third stage the Symbol CNN, as it can recognize words at
a symbol level. Figure 1 pictorially shows the dual stream architecture used.
The top stream takes in an image of size $32\times 128$ which is passed
through a series of convolutional layers and a fully connected layer. The
fully connected output (fc) size of the top stream is 1024. These features
represent the global information in the word block. The second stream consists
of only convolutional and pooling layers. The fully connected layer is
replaced by a fully convolutional layer which makes $2N+1$ predictions for a
word block of length $N$. The outputs from both the streams are added and
passed through a fully connected layer of size $N_{s}$ where $N_{s}$
represents the lexicon of symbols. The final output size is $(2N+1)\times
N_{s}$. We found that this dual stream model is able to encode rich
information about the global and local context of the symbols in the word
block.

$2N+1$ predictions were chosen such that each of the $N$ symbols in a word
block are straddled on both sides by a small gap which we refer to as a blank
space character. In particular we align the ground truth symbols with blank
spaces as shown in Figure 2. In our experiments, we found this to be intuitive
and empirically performs well in both dictionary constrained and non-
dictionary use cases. There are a total of 123 ground truth symbols which
include alphabets (English, French, Spanish, German), numbers, and special
characters such as $, &, period etc.

By not constraining the Symbol CNN output, our model is able to predict any
random sequence of characters such as phone numbers, social security numbers,
email ID etc. As such, the Symbol CNN is particularly useful to digitize tax,
insurance, and medical documents. The dual stream and full convolutional
nature of the model enables it to be quite effective at symbol recognition
with or without a word lexicon.

##  IV CER and Vocabulary Matching

Figure 2: During training, blank space characters(<b >) are added in between
ground truth symbols to improve symbol alignment.

Figure 3: Calculating CER error between words tymme and time using dynamic
programming. From left to right: after one step, after finishing “t”, after
finishing first “m”, and final CER of ’2’.

We report our results using Character Error Rate (CER):

$CER=R+D+C$ (1)

where $R$ is number of characters replaced, $D$ is the number of characters
deleted and $C$ is the number of correct characters. This can be efficiently
computed using dynamic programming and we show an example of the computation
in Figure 3. Equation 2 describes the CER computation where $C_{0,0}=0$ and
CER = $C_{l,h}$ where $l$ is the length of the label and $h$ is the length of
the prediction. $p_{i}$ is the $i^{th}$ character of the prediction and
$L_{i}$ is the $j^{th}$ character of the label.

$C_{i,j}=\textit{min($C_{i-1,j}$+{1}, $C_{i,j-1}$+{1}, Diag)}$ (2)

where:

$Diag=\begin{cases}C_{i-1,j-1},&if\hskip 2.84526ptp_{i}=L_{j}\\\
C_{i-1,j-1}+1,&otherwise\\\ \end{cases}$

To improve performance in applications that have a known-limited vocabulary,
we applied a CER-based vocabulary matching system using dynamic programming as
shown in (3),

$W(p)=\underset{L{\in}V}{\textrm{arg min }}\textit{CER(p,L)}$ (3)

where $V$ is the vocabulary set and ${W(p)}$ is the word prediction based on
the sequence prediction $p$. The sequence prediction $p$ is the set of symbols
predicted by FCN.

The above method improved CER, but discards most of the information computed
with the neural network. An improvement to the above vocabulary matching
system, which we refer to as probabilistic CER, uses character probabilities
instead of the discrete top character prediction. EquationIV describes
probabilistic CER method.

$\displaystyle C_{i,j}=\textit{min}(C_{i-1,j}+1-P(p_{i}=L_{j}),$
$\displaystyle C_{i,j-1}+1-P(p_{i}=blank),$ (4) $\displaystyle
C_{i-1,j-1}+1-P(p_{i}=L_{j}),)$

Since we now have a method to compute word probabilities from sequence
probabilities, we can combine predictions into the same system. We use the
frequency of occurrence of a given word $C(L)$ to further improve the
vocabulary matching using (5).

$W(p)=\underset{L{\in}V}{\textrm{arg min }}\textit{CER(p,
L)}+\frac{1}{1+C(L)}$ (5)

##  V Attention Modeling

When known variation of handwriting is both common and easily computer
generated, data augmentation can be used to generate a family of resized word
blocks to predetermined conditions. For example, given an input handwritten
word, deslanting algorithms can be used to create a family of inputs, each at
a predetermined slant. A similar family of inputs can be created by varying
stroke width, noise variation, and paper/background. Given a family of resized
word blocks, a vector of attention weights associated with the family of
resized word blocks is learned. Using the generated vector attention weight, a
single resized word block is formed as a linear combination of resized word
blocks, and passed through the CNN in normal fashion.

##  VI Training

We used Caffe [17] to perform all experiments. Custom layers were implemented
in python to handle ground truth and the blankspace symbol alignment. We
trained jointly on IAM [20], RIMES [3] and NIST [1] datasets. All experiments
used batch size of 64 (24 IAM, 24 RIMES and 16 NIST samples) and ran for 50k
iterations. We used a learning rate of 0.001 to train the network with 0.9
momentum. The learning rate was decreased to 0.0001 after 40k iterations. We
also used L2-regularization with $\lambda$ = 0.0025.

##  VII Results

Results are demonstrated on the IAM, RIMES, and NIST offline handwritten
datasets. The IAM [20] dataset contains 115,320 English words, mostly cursive,
by 500 authors. This dataset includes training, validation, and test splits,
where an author contributing to a training set, cannot occur in the validation
or test split. The RIMES [3] dataset contains 60,000 French words, by over
1000 authors. There are several versions of the RIMES dataset, where each
newer release is a super-set of prior releases. We utilize the popular
ICDAR2011 release.

The NIST Handprinted Forms and Characters Database, Special Database 19 [1],
contains NIST”s entire corpus of training materials for handprinted document
and character recognition. Each author filled out one or more pages of the
NIST Form-based Handprint Recognition System. It publishes handprinted sample
forms from 810,000 character images, by 3,600 authors.

###  VII-A IAM Results

We first test our system on the IAM English handwritten dataset. We fine-tuned
our model on IAM and it achieves CER of 4.43%. Table I highlights the results
of our model with dictionary correction which is quite competitive to the
current leaders of this dataset. Table II shows examples of predictions
obtained on the IAM dataset using only the FCN model without dictionary
correction.

TABLE I: Comparison of results on IAM dataset to previous methods. Model | WER | CER
---|---|---
Dreuw et al. [10] | 18.8 | 10.1
Boquera et al. [11] | 15.5 | 6.90
Kozielski et al. [18] | 13.30 | 5.10
Bluche et al. [5] | 11.90 | 4.90
Doetsch et al. [9] | 12.20 | 4.70
Our work | 8.71 | 4.43
Voigtlaender et al. [30] | 9.3 | 3.5
Poznanski and Wolf [23] | 6.45 | 3.44

Drewu et al. [10] showed that competitive results can be obtained by hybrid
approaches of MLP and Gaussian HMMs. Kozielski et al. [18] used a novel HMM
based system. Drewu et al. [10] and Boquera et al. [11] use a hybrid neural
network and Hidden Markov Model HMM approach. Bluche et al. [5] used Gaussian
HMMs to initialize neural networks and showed that both deep CNNs and RNNs
could produce state of the art results. Doetsch et al. [9] uses a custom LSTM
topology along with CTC alignment. [9, 5] used all words in a sentence and
paragraph respectively to provide word context. Poznanski and Wolf [23] used
deep CNNs to extract n-gram attributes which feed CCA word recognition. [18,
23, 9, 11] use deslanting, training augmentation, and an ensemble of test
samples.

Our work uses a first Vocabulary CNN of 1100 common words. The Symbol CNN uses
123 symbols, and we use probabilistic CER correction. Aside from the
probabilistic CER correction, no CTC alignment or CCA post correction was
applied. Although our competitive results are not ranked the best, our
processing path can work at both the symbol and lexicon level, and we include
substantially more symbols than prior methods (e.g. [23] can only recognize
upper and lower case Latin alphabet).

TABLE II: Symbol sequence prediction on the IAM dataset. The third example has a questionable ground truth and a prediction that could be considered valid out of context. Input | Label | Prediction
---|---|---
| that | that
| had | had
| Liverpool | livepool
| on | oui
| mistaken | mistahon
| ’ | ,
| implements | implement
| least | least
| mist | mist
| interest | interest

###  VII-B RIMES Results

We use a vocabulary CNN of 800 common words and fine-tuned our model on RIMES
dataset. Table III shows our model obtained a CER of 2.22% using an ensemble
of three models, each with dictionary correction. Table IV shows examples of
predictions obtained from one of the models without dictionary correction. In
general, errors can be attributed to character ambiguity, segmentation
artifacts (sample “effet” contains a comma even though it isn't part of the
label). For most of the examples in Table IV, our model made perfect
predictions.

TABLE III: Comparison of results on RIMES dataset to previous methods. Model | WER | CER
---|---|---
Kozielski et al. [18] | 13.70 | 4.60
Doetsch et al. [9] | 12.90 | 4.30
Bluche et al. [5] | 11.80 | 3.70
Our work | 5.68 | 2.22
Poznanski and Wolf [23] | 3.90 | 1.90
TABLE IV: Symbol sequence prediction on the RIMES dataset.  Input | Label | Prediction
---|---|---
| vous | vous
| titre | titre
| avancé | avance
| effet , | effett
| désire | diésiire
| téléphone | télénhone
| relevés | relves
| salutations | salutations
| l’expression | l’expression
| effectuer | effectuer
TABLE V: Symbol sequence prediction on NIST generated samples. Input | Label | Prediction
---|---|---
| 9/10/1966 | 9/10/1966
| (246)344-9702 | (246)344-9702
| $8643133 | $8643133
| Spectrometry | Spectrometry
| +6091620 | +6091620
| 92.84.8A.b4.AE.15 | 92.84.8A.b4AE.15

###  VII-C NIST

While there are several class specific handwritten datasets, both at the
character and word level, there is no large handwritten dataset that
concentrates on word blocks of arbitrary symbols. To test the performance of
our model on generic word blocks made of arbitrary symbols, we created a new
symbol recognition dataset by stochastically combining the NIST individual
character images into realistic word blocks. Images of hand printed text are
simulated by extracting character images from a randomly selected writer in
the NIST dataset and concatenating them into word blocks of random dictionary
words, random strings of alphanumeric characters, or random strings of numeric
characters. In addition, the NIST dataset has been supplemented with
handwritten punctuation, mathematical symbols and other common special
characters such as the dollar sign, email character and the ampersand to
facilitate in generating word block images of common form-field inputs.

The images are further augmented by adding random amounts of displacement,
stretching and rotation to each symbol to simulate the natural variability in
writer’s penmanship. A random amount of skew is then applied to each
concatenated image to vary the slant of the word block. Finally, random
amounts of noise and blur are added to simulate the effects of image
digitization. Table V shows randomly generated samples from NIST characters
and corresponding predictions. Our model achieves 92.4% accuracy on a subset
of 12,000 word blocks (consisting English, French and special characters)
generated from NIST.

Some of the characters in handwritten word blocks are too hard to decipher and
require contextual information for human or machine interpretation. In order
to provide more contextual information and recognize characters which are
ambiguous due to incomplete pen strokes and inconsistencies in handwriting, we
utilized both the contextual path introduced at the top of Figure 1 as well as
an attention mechanism. After preprocessing an input word block with a series
of known variations such as slant and noise, the attention mechanism
formulates a single input as a linear combination of these variations.
Although the attention mechanism is intuitive, empirical results have yet to
show any significant improvement and were not used in the results in Tables I
or III.

##  VIII Conclusion

We introduce a novel offline handwriting recognition algorithm using a fully
convolutional network. Unlike lexicon constrained methods, our method can
recognize common words as well as infinite symbol blocks such as surnames,
phone numbers and acronyms. Our dual stream architecture along with blank
space symbol alignment eliminates the need of complex character alignment
methods such as CTC in recurrent based methods. Our fully convolutional model
enables processing of arbitrary length inputs and utilizes a large symbol set.
Despite not using a word lexicon and handling of a large symbol set, our
method performs on par with current state of the art methods on English based
IAM, French based RIMES, and NIST arbitrary symbol handwritten dataset. Our
model is flexible and can be easily extended to other languages.

## References

  * [1] Nist special database 19, 2nd ed.  Accessed: 2017.
  * [2] Saad Bin Ahmed, Saeeda Naz, Muhammad Imran Razzak, and Rubiyah Yousaf.  Deep learning based isolated arabic scene character recognition.  In Arabic Script Analysis and Recognition (ASAR), 2017 1st International Workshop on, pages 46–51. IEEE, 2017.
  * [3] Emmanuel Augustin, Matthieu Carré, Emmanuèle Grosicki, J-M Brodin, Edouard Geoffrois, and Françoise Prêteux.  Rimes evaluation campaign for handwritten mail processing.  In International Workshop on Frontiers in Handwriting Recognition, pages 231–235, 2006.
  * [4] Batuhan Balci, Dan Saadati, and Dan Shiferaw.  Handwritten text recognition using deep learning.  CS231n: Convolutional Neural Networks for Visual Recognition, Stanford University Project, 2017.
  * [5] Théodore Bluche, Hermann Ney, and Christopher Kermorvant.  A comparison of sequence-trained deep neural networks and recurrent neural networks optical modeling for handwriting recognition.  In International Conference on Statistical Language and Speech Processing, pages 199–210. Springer, 2014.
  * [6] Francesca Cesarini, Marco Gori, Simone Marinai, and Giovanni Soda.  Structured document segmentation and representation by the modified xy tree.  In Proceedings of the Fifth International Conference on Document Analysis and Recognition, pages 563–566, 1999.
  * [7] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  arXiv preprint arXiv:1606.00915, 2016.
  * [8] Sagar Dewan and Srinivasa Chakravarthy.  A system for offline character recognition using auto-encoder networks.  In International Conference on Neural Information Processing, pages 91–99. Springer, 2012.
  * [9] Patrick Doetsch, Michal Kozielski, and Hermann Ney.  Fast and robust training of recurrent neural networks for offline handwriting recognition.  In 14th International Conference on Frontiers in Handwriting Recognition, pages 279–284, 2014.
  * [10] Philippe Dreuw, Patrick Doetsch, Christian Plahl, and Hermann Ney.  Hierarchical hybrid mlp/hmm or rather mlp features for a discriminatively trained gaussian hmm: a comparison for offline handwriting recognition.  In 18th IEEE International Conference on Image Processing, pages 3541–3544, 2011.
  * [11] Salvador Espana-Boquera, Maria Jose Castro-Bleda, Jorge Gorbe-Moya, and Francisco Zamora-Martinez.  Improving offline handwritten text recognition with hybrid hmm/ann models.  IEEE transactions on pattern analysis and machine intelligence, 33(4):767–779, 2011.
  * [12] Paul D Gader, Magdi Mohamed, and Jung-Hsien Chiang.  Handwritten word recognition with character and inter-character neural networks.  IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 27(1):158–164, 1997.
  * [13] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks.  In Proceedings of the 23rd international conference on Machine learning, pages 369–376. ACM, 2006.
  * [14] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [15] Chen Huang and Sargur N Srihari.  Word segmentation of off-line handwritten documents.  In Document Recognition and Retrieval XV, volume 6815, page 68150E. International Society for Optics and Photonics, 2008.
  * [16] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Synthetic data and artificial neural networks for natural scene text recognition.  arXiv preprint arXiv:1406.2227, 2014.
  * [17] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell.  Caffe: Convolutional architecture for fast feature embedding.  In Proceedings of the 22nd ACM international conference on Multimedia, pages 675–678. ACM, 2014.
  * [18] Michał Kozielski, Patrick Doetsch, and Hermann Ney.  Improvements in rwth’s system for off-line handwriting recognition.  In Document Analysis and Recognition (ICDAR), 2013 12th International Conference on, pages 935–939. IEEE, 2013.
  * [19] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [20] U-V Marti and Horst Bunke.  The iam-database: an english sentence database for offline handwriting recognition.  International Journal on Document Analysis and Recognition, 5(1):39–46, 2002.
  * [21] Farès Menasri, Jérôme Louradour, Anne-Laure Bianne-Bernard, and Christopher Kermorvant.  The a2ia french handwriting recognition system at the rimes-icdar2011 competition.  In Document Recognition and Retrieval XIX, volume 8297, page 82970Y. International Society for Optics and Photonics, 2012.
  * [22] Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour.  Dropout improves recurrent neural networks for handwriting recognition.  In 14th International Conference on Frontiers in Handwriting Recognition, pages 285–290, 2014.
  * [23] Arik Poznanski and Lior Wolf.  Cnn-n-gram for handwriting word recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2305–2314, 2016.
  * [24] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, pages 91–99, 2015.
  * [25] Baoguang Shi, Xiang Bai, and Cong Yao.  An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition.  IEEE transactions on pattern analysis and machine intelligence, 39(11):2298–2304, 2017.
  * [26] Baoguang Shi, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai.  Robust scene text recognition with automatic rectification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4168–4176, 2016.
  * [27] Carlos N Silla and Celso AA Kaestner.  An analysis of sentence boundary detection systems for english and portuguese documents.  In International Conference on Intelligent Text Processing and Computational Linguistics, pages 135–141. Springer, 2004.
  * [28] Zenghui Sun, Lianwen Jin, Zecheng Xie, Ziyong Feng, and Shuye Zhang.  Convolutional multi-directional recurrent network for offline handwritten text recognition.  In 15th International Conference on Frontiers in Handwriting Recognition, pages 240–245, 2016.
  * [29] Øivind Due Trier, Anil K Jain, and Torfinn Taxt.  Feature extraction methods for character recognition-a survey.  Pattern recognition, 29(4):641–662, 1996.
  * [30] Paul Voigtlaender, Patrick Doetsch, and Hermann Ney.  Handwriting recognition with large multidimensional long short-term memory recurrent neural networks.  In 15th International Conference on Frontiers in Handwriting Recognition, pages 228–233, 2016.
  * [31] Zecheng Xie, Zenghui Sun, Lianwen Jin, Ziyong Feng, and Shuye Zhang.  Fully convolutional recurrent network for handwritten chinese text recognition.  In 23rd International Conference on Pattern Recognition, pages 4011–4016, 2016.
  * [32] Usha Yadav, Satya Verma, Deepak Kumar Xaxa, and Chandrakant Mahobiya.  A deep learning based character recognition system from multimedia document.  In Power and Advanced Computing Technologies (i-PACT), 2017 Innovations in, pages 1–7. IEEE, 2017.