# Automatic Product Copywriting for E-Commerce

Xueying Zhang1, Yanyan Zou2, Hainan Zhang2, Jing Zhou2, Shiliang Diao2, Jiajia
Chen2, Zhuoye Ding2, Zhen He2, Xueqi He2, Yun Xiao1, Bo Long2, Han
Yu3$\dagger$, Lingfei Wu1$\dagger$

###### Abstract

Product copywriting is a critical component of e-commerce recommendation
platforms. It aims to attract users’ interest and improve user experience by
highlighting product characteristics with textual descriptions. In this paper,
we report our experience deploying the proposed Automatic Product Copywriting
Generation (APCG) system into the JD.com e-commerce product recommendation
platform. It consists of two main components: 1) natural language generation,
which is built from a transformer-pointer network and a pre-trained sequence-
to-sequence model based on millions of training data from our in-house
platform; and 2) copywriting quality control, which is based on both automatic
evaluation and human screening. For selected domains, the models are trained
and updated daily with the updated training data. In addition, the model is
also used as a real-time writing assistant tool on our live broadcast
platform. The APCG system has been deployed in JD.com since Feb 2021. By Sep
2021, it has generated 2.53 million product descriptions, and improved the
overall averaged click-through rate (CTR) and the Conversion Rate (CVR) by
4.22% and 3.61%, compared to baselines, respectively on a year-on-year basis.
The accumulated Gross Merchandise Volume (GMV) made by our system is improved
by 213.42%, compared to the number in Feb 2021.

## Introduction

Figure 1: Overview of the proposed Automatic Product Copywriting Generation
(APCG) system.

With the rapid development of the Internet, online shopping has become
indispensable in people’s daily life. E-commerce is now a mainstay way of
shopping for many people all over the world. Product advertising copywriting
plays an important role in e-commerce platforms, where well-written
advertisements can help attract customers and, in turn, increase sales.
Traditionally, product copywriting is performed by human copywriters. This
approach has important limitations. Firstly, the efficiency of human
copywriters cannot match the growth rate of new products. Secondly, manpower
cost increases as the number of products in the system increases. Last but not
least, specialist training and tutoring is required for copywriters for
different products and scenarios. To overcome these issues, automatic product
copywriting generation has become an important line of research in e-commerce.

Natural language generation (i.e., NLG) research focuses on the construction
of computer systems that can produce understandable texts in human languages
from some underlying non-linguistic or textual representation of information
(Reiter and Dale 2000). NLG systems combine knowledge from both the language
and the application domain to automatically generate desired texts. In recent
years, NLG techniques have been developing rapidly and applied to a wide range
of applications. For instance, abstract text summarization (Rush, Chopra, and
Weston 2015; Banko, Mittal, and Witbrock 2000; Knight and Marcu 2000)
generates a summarized version of input text to highlighted certain
information; dialogue generation (Tao et al. 2018; Zhang et al. 2020)
generates responses according to the previous chats; machine translation
(Bahdanau, Cho, and Bengio 2016) translates the input texts into a different
language; and storytelling or poem generation (Yao et al. 2019; Li et al.
2018) generates creative texts.

In the early stage of text generation research, due to limitation of data and
computational resources, models relied heavily on feature engineering. In this
case, features are extracted from raw data based on human knowledge and
preference, and fed into the models to learn from the limited data (Lafferty,
McCallum, and Pereira 2001; Och et al. 2004). With the development of deep
learning, features are learned automatically and defined implicitly during
model training. The natural language generation problem has been framed as a
sequence-to-sequence task and various studies have been proposed to enhance
the model architecture, such as convolutional neural networks (CNN) (Gehring
et al. 2017), recurrent neural networks (RNN) (Sutskever, Vinyals, and Le
2014), graph neural networks (GNN) (Wu et al. 2020), and attention-based
architecture (Bahdanau, Cho, and Bengio 2016; Vaswani et al. 2017) (especially
the transformer network (Vaswani et al. 2017)).

Recently, the pre-training plus fine-tuning paradigm has gained traction and
been widely applied in real-world applications. Under this paradigm, models
are first pre-trained on large-scale corpus (Devlin et al. 2019a; Brown et al.
2020), or on domain-specific corpus (Zhang et al. 2021) with well-defined pre-
training tasks. The resulting pre-trained models are then fine-tuned to adapt
to different downstream tasks. Research in this paradigm mainly focuses on
defining training objectives in the pre-training and fine-tuning stages. With
increasing computational resources and the emerging transformer architecture
(Vaswani et al. 2017), recent studies (e.g., GPT2 (Radford et al. 2018b), CTRL
(Keskar et al. 2019), UNiLM (Dong et al. 2019)) utilizing pre-training and
fine-tuning techniques have achieved outstanding performance in language
generation tasks.

In this paper, we report on our experience deploying the proposed Automatic
Product Copywriting Generation (APCG) system into the
JD.com111https://www.jd.com/ e-commerce product recommendation platform. It is
made up of two main components: 1) a natural language generation (NLG) module,
which is built from a transformer-pointer network and a pre-trained sequence-
to-sequence model; and 2) a copywriting quality control module, which involves
both automatic evaluation and human screening to ensure the quality of the
generated descriptions. For selected domains, the models are trained and
updated daily with the updated training data. In addition, the model is also
used as a real-time writing assistant tool on our live broadcast platform.

Since its deployment in JD.com in Feb 2021, APCG has generated 2.53 million
product descriptions by Sep 2021. As a result, the overall click-through rate
(CTR) has been improved by 4.22%, the conversion rate (CVR) has been improved
by 3.61%, and the accumulated gross merchandise volume (GMV) has more than
doubled on a year-on-year basis.

## Application Description

An online e-commerce website typically describes a product $p$ from several
aspects, including the product title $t$, a set of the product attribute $A$
and the corresponding attribute values $V$ for each attribute $a\in A$, as
well as the short advertisement slogan $s$ written by advertising experts. The
product title $t$ usually describes the product with a short text, expressing
the key information, such as the brand name and product type. The product
attribute set $A$ captures the properties of a product from different
perspectives. For example, the product functions, intended users, styles,
colors and materials. The corresponding values of the attributes from $A$ form
the product attribute value set $V$. For example, the product function feature
can be _breathable, resistant_ , and the intended user feature can be
_ladies_. Given the title $t$, attribute set $A$, the corresponding attribute
value set $V$ and the advertisement slogan $s$, the task of product
copywriting is to generate a well-written description $D$ that is able to
present product characteristics for the e-commerce system to attract user
interest and keep them informed of the key features of the product quickly. An
example of product copywriting is demonstrated in Figure 4.

Early work on product copywriting applied template-based generation methods
with statistical knowledge extracted from the training data (Wang et al.
2017), while the template coverage and diversity are limited by the training
corpus. Chen et al. 2019 designed a transformer-based generation model to
incorporate the external knowledge (i.e., Wikipedia knowledge base). Besides,
the textual information (i.e., product title and attributes) and the product
image features are also incorporated. The performance of such a model heavily
relies on the availability of related knowledge. Similarly, Li et al. 2020
constructed a set of aspects (i.e., salient attributes) and the corresponding
keywords for each aspect, which, however, requires expert annotations. An
adaptive posterior Transformer-based network (Zhan et al. 2021) has been
designed to utilize relevant information from customer reviews extracted by an
adaptive posterior distillation module, where the product title and attribute
information is considered as the main model input.

However, neither the external knowledge base nor the customer reviews are
always accessible. In practice, for the e-commerce platform, newly released
products are emerging every day, where the product information is quite
limited. In addition, such kind of external features may contain noise,
especially for newly released products. This reduces the quality of the
generated product descriptions. On the other hand, each product is accompanied
with several short advertising phrases written by experts. However, existing
studies did not leverage this information when generating product
descriptions. In this work, we jointly consider the product textual features
(i.e., title, attribute and advertising slogan) as product input information,
and develop a transformer-pointer model and a pre-trained sequence-to-sequence
model for automatic product copywriting in e-commerce settings. Both of such
two models constitute our Automatic Product Copywriting Generation (APCG)
system.

Figure 2: Workflow of the APCG system.

As illustrated in Figure 1, our proposed APCG system is designed to serve a
variety of applications in the JD.com e-commerce platform, including the
“HaoHuo” (referring to “Discovery Goods”) product recommendation platform, JD
books, searching, global shopping, the “WoZhe” platform (mainly for home
appliance) and the live broadcast selling platform, etc. As shown in Figure 2,
the overall system consists of the following steps:

  * 1)

NLG model training: To collect data, including both product information and
descriptions, for training the NLG model. Different models are selected upon
different scenarios. Both the transformer-pointer network and the pre-trained
language models are applied in this stage.

  * 2)

Post-evaluation for direct customer use: Post-processing models are utilized
here to filter out contents which do not meet the requirements. Human
screening is performed after model-based filtering. The contents produced here
are directly used by the product recommendation platform for JD customers.

  * 3)

Real-time writing assistant tool for sellers: Besides generating and filtering
contents based on proposed product skus, a real-time writing assistant tool
based on APCG has been developed for JD sellers. With this tool, JD sellers
can input their desired products, and it will automatically generate product
descriptions. Sellers can then edit the generated contents, and display them
to customers.

  * 4)

Automatic procedure with daily update: APCG is trained daily to meet the
latest writing preference based newly updated data.

## Use of AI Technology

In this section, we describe the key techniques adopted by the APCG AI Engine.

### Data Collection and Cleaning

#### Data Collection

We built a large-scale real-world dataset from our in-house platform which
includes two parts: 1) product description data, and 2) product information
data. The product descriptions are written by professional copywriters with
good knowledge of marketing. For each product, the description includes both
advertising contents and the title. For the product information data, we
collected multiple types of data from our in-house database, including product
titles, attributes, product detail images and user reviews.

#### Data Cleaning

After obtaining the large-scale datasets, we perform filtering and cleaning on
the raw data to construct our training dataset. The product descriptions are
filtered by human evaluation and rule-based methods, and used as the outputs
of our model. The inputs include information from product titles, attributes,
product detail images and user reviews. For each category, we select different
groups of attributes which have high relevance with the target products. For
product detail images, we leverage the optical character recognition (OCR) and
classification techniques to extract key information about the product.
Firstly, texts are extracted from the product detail images. Then, the
extracted texts are ranked in descending order of their importance and
relevance using the classification model. Finally, highly ranked texts are
selected and merged as the final OCR input. We use the ranking model to select
useful reviews, and a summarization model to obtain highlights of selected
reviews, which are then used as additional inputs.

### The Transformer-Pointer Network

Based on this large-scale proprietary dataset, the automatic product
copywriting problem can be regarded as an abstractive summarization problem.
We designed a pointer-generator network, which allows the transformer network
to generate new words and copy words form the input text at the same time.
Through this transformer-pointer network, issues of inaccurate descriptions
and unknown words can be handled since both extractive and abstractive
approaches are executed simultaneously.

The proposed transformer-pointer network is a combination of a pointer-
generator network and a transformer structure. As shown in Figure 3, by
introducing the trainable probability $P_{gen}$, the model learns to choose
whether to use words from the vocabulary source or from the input text. APCG
will select a new word from the vocabulary distributions with a probability of
$P_{gen}$, and copy words directly from the input text through the selective
pointer with a probability of $(1-P_{gen})$. We take the encoder-decoder
attentions in the last decoder layer as the copy distribution. Note that for
the multi-head attention, we obtain the copy distributions by summing across
multiple heads.

Figure 3: The transformer-pointer network structure.

### The Pre-trained Language Model

Training a neural network from scratch requires large amounts of data with
high-quality labels, especially true for the generation task. Unlabeled open-
domain data (i.e., without gold standard annotations) are readily available
from online sources such as Wikipedia. Recently, pre-trained models, such as
OpenAI GPT (Radford et al. 2018a, 2019), BERT (Devlin et al. 2019b), RoBERTa
(Liu et al. 2019) and BART (Lewis et al. 2019), have become popular for NLP
tasks. They have significantly improved the state-of-the-art performance on
various natural language understanding (NLU) tasks (e.g., SQuAD question
answering (Rajpurkar et al. 2016), natural language inference (Bowman et al.
2015), text classification (Zhang, Zhao, and LeCun 2015)) and natural language
generation tasks (e.g., machine translation (Song et al. 2019) and text
summarization (Zou et al. 2020)). This demonstrates the effectiveness and
transferability of models pre-trained on large scale unlabeled data. Recent
research progresses in (Brown et al. 2020) have demonstrated that pre-trained
models can play as an important role in few-shot learning with limited
training instances.

In the product copywriting scenario, there are different categories of
products (e.g., clothing, mobile phones, sportswear). The product
characteristics and data properties vary significantly across different
categories. One straightforward solution is to train the transformer-pointer
model on the training data for each category individually. However, this will
incur high computation and maintenance costs. In addition, as the products in
the e-commerce are updated frequently, there is often limited or even no
training data for newly released products in practice. Therefore, we adopt the
pre-trained model for product copywriting.

We adopt the sequence-to-sequence transformer network (Vaswani et al. 2017),
which comprises of an encoder and a decoder, as the backbone architecture.
Given an input token sequence $X=(x_{1},x_{2},\dots,x_{|X|})$ paired with its
corresponding output sequence $Y=(y_{1},y_{2},\dots,y_{|Y|})$, we aim to learn
the model parameters $\theta$ and estimate the conditional probability:

$\vspace{-0mm}P(Y|X;\theta)=\prod_{t=1}^{|Y|}p(y_{t}|y_{<t};X;\theta)$ (1)

where $y_{<t}$ stands for all tokens before position $t$ (i.e.,
$y_{<t}=(y_{1},y_{2},\dots,y_{t-1})$). Given the whole training set
$(\mathcal{X},\mathcal{Y})$, this model can be trained by maximizing the log-
likelihood of the training input-output pairs:

$\vspace{-0mm}\mathcal{L}(\theta;\mathcal{X},\mathcal{Y})=\sum_{(X,Y)\in(\mathcal{X},\mathcal{Y})}\log
P(Y|X;\theta).$ (2)

We pre-train the model using sequence-to-sequence objectives.

Recall that the attributes of a product are represented by the set $A$ without
ordering information. We observe that the transformer model generates
different outputs when receiving attributes in different orders as inputs.
This can sometimes produce poor quality outputs. On the other hand, given the
product title, attributes and advertising slogans, the task of product
copywriting is to generate the description, where the generated text contains
tokens copied from input text and novel words and phrases that are not
contained in the input. Based on these observations, we design two sequence-
to-sequence objectives for pre-training.

##### Sentence Re-ordering

Inspired by (Zou et al. 2020), we incorporate the sentence re-ordering
objective (SR) in APCG. We first divide an unlabeled document into multiple
sentences based on full stop signs. For better demonstration, we change the
notation of a document slightly in this paragraph. Let
$X=(S_{1}||S_{2}||\dots||S_{m})$ denote a document, consisting of several
sentences, where $S_{i}$ is a sentence, $m$ is the number of sentences, and
$||$ refers to sentence concatenation. The sentence index order in $X$ can be
represented as $\mathcal{O}=(1,2,\dots,m)$. We then shuffle the sentences in
the document. In other words, the items in the order $\mathcal{O}$ are re-
arranged and we obtain a shuffled order
$\mathcal{O}_{S}=(a_{1},a_{2},\dots,a_{m})$, where $a_{i}$ refers to sentence
index, $1\leq a_{i}\leq m$, $1\leq a_{j}\leq m$, and $a_{i}\neq a_{j}$ for any
$i,j\in[1,m]$ and $i\neq j$. Concatenating sentences following
$\mathcal{O}_{S}$, we obtain a _shuffled_ document
$\hat{X}_{S}=(S_{a_{1}}||S_{a_{2}}||\dots||S_{a_{m}})$. The sequence-to-
sequence model takes the shuffled document $\hat{X}_{S}$ as input and is pre-
trained to re-instate the original one $X$. The training objective is
expressed as:

$\vspace{-0mm}\mathcal{L}(\theta;\mathcal{X})=\sum_{X\in\mathcal{X}}\log
P(X|\hat{X}_{S};\theta).$

##### Pseudo Summary Generation

This objective is adopted from (Su 2021). Suppose a document is denoted as
$X=(S_{1}||S_{2}||\dots||S_{m})$. We select around $m/4$ sentences from the
document $X$, and concatenate into $\hat{X}_{M}$. The remaining $3m/4$
sentences are concatenated into $X_{R}$. The sentences in $\hat{X}_{M}$ are
carefully selected so that $\hat{X}_{M}$ and $X_{R}$ contain the longest
subsequences among all alternatives. The longer text $X_{R}$ is considered as
the model input, while the shorter text $\hat{X}_{M}$ is regarded as the model
output. Thus, the training objective is expressed as:

$\vspace{-0mm}\mathcal{L}(\theta;\mathcal{X})=\sum_{X\in\mathcal{X}}\log
P(X_{R}|\hat{X}_{M};\theta).$

After a sequence-to-sequence model is pre-trained, we fine tune the model on
the parallel copywriting training data, consisting of instances from all
categories.

### Post-Processing

Although the proposed language generation models can generate texts with good
quality most of the time, there is still possibility that the generated
descriptions might not be accurate. Thus, we incorporate post-processing
models into APCG to further enhance copywriting quality to meet the high
expectation in practical e-commerce applications.

In some cases, the descriptions generated by the model are not consistent with
product information. For example, for iPhone 11, the model generates the
product description of “iPhone 11采用升降式摄像头” (iPhone 11 with a foldable webcam),
which is not an actual feature of iPhone 11. Although cases like this are
rare, they can negatively affect user experience. To overcome this issue, we
designed a rule-based product-word and number checking model. We collect words
and numbers related for each product item, and construct our specific
e-commerce domain specific word dictionary. We develop rule-based models for
each product category based on these dictionaries to filter out generated
descriptions with inaccurate information. The rules are based on combinations
of terms, where single words within them are not affected.

Although NLG models are trained to achieve good fluency and grammar
correctness, they are not fool proof, especially when the input information is
too new for the model. To address this limitation, we train grammar checking
models and leverage the Adaboost algorithm to deploy these models to filter
out generated descriptions with poor grammar quality. These post-processing
approaches help APCG filter out poorly generated description and enhance the
copywriting quality.

## Application Development and Deployment

### Offline Evaluation

We compare our transformer-pointer model (TP) and pre-training based
copywriting generation model (PCG) with two baselines before making deployment
decisions:

  * •

The template-based generation model (i.e., Template) (Wang et al. 2017); and

  * •

The attribute mining based generation model (i.e., A-Mining) (Li et al. 2020).

For fair comparison, we randomly selected 500 products from different
categories, and generated descriptions using the four models. We used
SacreBLEU (Post 2018), ROUGE (Lin 2004), BLEU (Papineni et al. 2002), and
Meteor (Lavie, Sagae, and Jayaraman 2004) to measure the quality of different
generation model outputs. We report the SacreBLUE, ROUGE-1, ROUGE-2, ROUGE-L,
BLEU-1, BLEU-2, BLUE-3, BLEU-4 and Meteor scores in Table 1. The larger the
values of these metrics, the better the performance of a model. It can be
observed that PCG consistently achieves the best performance under all
evaluation metrics.

Since descriptions generated by models may produce incoherent or grammatically
incorrect outputs, we also evaluated these four models by eliciting human
judgements. Ten human participants are presented with a product and a list of
descriptions generated by the four different models in random orders. Then,
they are asked to rank these descriptions according to fluency (_is the
description grammatically sound?_), relevance (_is the description relevant to
the product?_), and diversity (_does the text describe the product from
different perspectives?_) The results are listed in Table 2. It can be
observed that PCG achieves the best performance under all human evaluation
metrics. These results helped us make the decision to incorporate TP and PCG
into the APCG system.

Table 1: Offline evaluation results for the four models.

Model | SacreBLUE | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | Meteor
---|---|---|---|---|---|---|---|---|---
Template | 4.84 | 14.20 | 2.65 | 11.01 | 27.95 | 15.94 | 10.09 | 6.80 | 12.72
A-Mining | 5.35 | 16.33 | 3.43 | 13.04 | 30.24 | 17.31 | 10.83 | 7.27 | 13.57
TP | 5.81 | 17.37 | 4.14 | 14.06 | 29.80 | 17.90 | 11.70 | 8.17 | 13.77
PCG | 6.19 | 18.98 | 4.37 | 15.66 | 30.29 | 18.34 | 11.97 | 8.17 | 14.04

Table 2: Human evaluation results for the four models. Model | Fluency | Relevance | Diversity
---|---|---|---
Template | 1.78 | 1.89 | 1.79
A-Mining | 1.80 | 1.90 | 1.77
TP | 1.80 | 1.84 | 1.81
PCG | 1.96 | 1.91 | 1.87

### Deployment Usage

The procedure of automatic product copywriting generation includes the
following steps: 1) obtaining the required product skus from the database; 2)
generating product descriptions and performing post-processing; 3) submitting
the generated contents for human screening; 4) storing approved contents into
the database for customer service provision. Daily uploads to the descriptions
database can reach up to 28,570 generated descriptions, with an average daily
acceptance rate of 80%. The usage of the JD product recommendation platform
generates new data daily, our NLG models are trained and updated daily with
these new data.

Figure 4: Examples of APCG generated descriptions on the JD e-commerce product
recommendation platform. Figure 5: Examples of APCG generated descriptions on
the JD live broadcast selling platform. The virtual salesperson reads the
product descriptions in the virtual studio.

As illustrated in Figure 2, the deployed APCG system serves two main usage
scenarios: 1) content generation for customers, and 2) real-time writing
assistant for sellers. In the first scenario, the contents generated by APCG
are stored in our database and displayed directly on the JD product
recommendation platform. Figure 4 shows an example of the product
recommendation platform for customers, “HaoHuo”. The English translations of
the generated texts are shown in the pop up bubbles in the figure. The
generated product descriptions, paired with a short title, is pushed to the
Discovery Goods Channel in the JD e-commerce website. The short title consists
of three phrases: 1) the brand name, 2) one of salient product
characteristics, and 3) the product name, which make up a product headline. To
further ensure the user experience, we also display three images for a
product. Overall, the short title, description and three images are combined
together to depict a product in the Discovery Goods Channel. APCG only focuses
on the description generation task.

In the second scenario, a real-time writing assistant tool is designed for
sellers, especially for sellers promoting products on the live broadcast
platform. The sellers can input the product ID they would like to advertise,
and the APCG writing assistant tool will generate product descriptions fitting
for the current context in real time. Figure 5 shows the use case in the live
broadcast selling platform. By Sep, 2021, the APCG writing assistant tool has
been adopted by more than 2,000 merchants with daily live streams up to 150
sessions.

As real-time performance is important for the second use case, we optimize the
design of APCG for this purpose. We split the encoder and decoder of our
transformer model, and design interfaces to realize the beam search algorithm.
The two main interfaces include the encoder predictor and decoder predictor.
The encoder predictor receives the input tokens, and returns the token IDs and
the encoded embedding. For the decoder predictor, the inputs include the token
IDs, the encoded embedding and the current prefix. It outputs the top-$k$
candidates of the predicted indices, tokens and probabilities. Due to language
generation problems, contextual words need to be generated one-by-one. Thus,
when generating descriptions for one item, the encoder interface is called
once and the decoder interface is called multiple times depending on the
length of the context and the beam size. For beam search, extra storage is
required for holding the candidates within the beam.

We conduct latency test for our real-time writing assistant tool. Table 3
shows that the GPU predictor can handle high query request traffic of up to
100 queries per second (QPS), which is four times the throughput of the CPU
predictor. The average latency and TP-99 (i.e., the minimum time under which
99% of requests have been served) achieved by using GPU are about 1/5 of using
CPU.

To further enhance the efficiency, before calling our language generation
engine to generate contents in the real time, we first check whether the
proposed input product is in our copywriting description database. If so, we
will directly provide the previously generated contents stored in the database
to save time.

Table 3: Latency test comparison of CPU and GPU services. Service | QPS | avg latency (ms) | TP-99 (ms)
---|---|---|---
CPU predictor | 23.0 | 80.0 | 106.0
GPU predictor | 100.0 | 15.6 | 19.2

## Maintenance

The key performance indicator used to monitor the performance of APCG is the
daily acceptance rate. If this rate becomes low, we will perform manual checks
on the rejected descriptions and fix the issues in the model accordingly.
Besides this, no other major maintenance task has been required since its
deployment in Feb 2021.

## Application Use and Payoff

The APCG system has been deployed in JD.com since Feb 2021. To demonstrate the
payoff generated by APCG, we compare key performance indicators before and
after deploying APCG on the JD Discover Goods Channel and its mobile app. A
standard A/B testing is used on such online platforms to evaluate the benefit
of the APCG-generated product descriptions for the business on the platform.
The impact on the business of the platform is measure by the Click-Through
Rate (CTR) and Conversion Rate (CVR). CTR is defined as:

$\text{CTR}=\frac{\text{\\#clicked\\_product}}{\text{\\#pv\\_product}}$ (3)

where click_product is the number of clicks on the products, and pv_product is
the number of page views for the product shown to the users. CVR is defined
as:

$\text{CVR}=\frac{\text{\\#converted\\_product}}{\text{\\#clicked\\_product}}$
(4)

where the converted_product refers to the number of times a product has been
purchased by users.

For online deployment, we randomly selected around 500,000 products covering
20 categories, and generated corresponding descriptions using both the
Transformer-Pointer and Pre-trained models. The baseline method is the
A-Mining model which was used by the platform prior to the deployment of APCG.
We first conduct A/B testing for the baseline model and Transformer-Pointer
model. During the online A/B testing period, over five million people visited
the Discovery Good Channel, providing us with adequate opportunities for
testing. The A/B testing system divided these visiting users into two equal-
sized groups, and directed them into two individual buckets (i.e., buckets A
and B). For users in bucket A (i.e., the baseline group), the system shows
them the product descriptions generated by the A-Mining model. For users in
bucket B (i.e., the experiment group), the system shows them the product
descriptions generated by the Transformer-Pointer model in APCG.

The CTR and CVR for all products during the A/B testing period are calculated.
We found that the our Transformer-Pointer model significantly outperforms the
previous method by a large margin. Specifically, the TP model improved the CTR
by 3.44% and CVR by 3.40% over the A-Mining model. The increase of the CTR
indicates that users prefer to click the products with descriptions generated
by our system. The improvement of CVR demonstrates that the descriptions
generated by our model is successful in convincing users to make purchases.

Another A/B test has been conduced with our pre-trained model (PCG). This
time, we use our Transformer-Pointer model as the baseline. All other A/B test
settings remain the same. Compared to the baseline, the CTR and CVR are
improved by a relatively small margin of 1.3% and 0.8%, respectively, which is
in line with expectation. Recall that, the Transformer-Pointer model is
deployed individually for each category, while the PCG model is able to
provide description generation services for all categories. However, the pre-
trained model incurs high computation costs. Thus, during APCG deployment, PCG
mainly focuses on categories with limited training data or newly released
products.

Between Feb 2021 and Sep 2021, APCG has generated 2.53 million product
descriptions for the JD.com e-commerce platform. The overall averaged CTR
achieved by APCG is 4.22% higher than that achieved by the previous system
year-on-year (yoy). The improvement in terms of CTR is 3.61% yoy. The
accumulated Gross Merchandise Volume (GMV) made by our system is improved by
213.42%, compared to the start number in Feb 2021. By introducing automated
copywriting, our daily uploads to the descriptions database can reach up to
28,570 generated descriptions, equivalent to the productivity of thousands of
human experts, which significantly saves human labor costs. Although the
product copywriting task in the JD Discover Goods Channel is currently jointly
performed by expert writers and the APCG systems, we observed that expert
writers often prefer to create descriptions for popular products. Differently,
our system (especially the pre-trained model) is able to cover both popular
and long-tail products, which helps build a healthy ecosystem. Thus, it is
expected that the AI system will take over this task completely in the near
future.

## Lessons Learned During Deployment

At the time of submission of this paper, APCG has been successfully deployed
for around six months and has led to significant positive impact on JD.com’s
business. Several lessons we have learned during model deployment could be
beneficial for other like-minded researchers and practitioners who wish to
deploy cutting-edge AI technologies into real-world applications.

Firstly, besides the model capacity, the quality of training data is of
paramount importance. The cleaning procedures of raw data (e.g., removing poor
samples from training set and specifying group of important attributes) play
an important role in model development. Today, data pre-processing is mainly
based on human experiences and rules, which is expensive and might not work
well in new domains. A possible alternative approach one can take is to design
a classification model that can identify low quality data from the training
set (e.g., through influence function-based approaches (Chen et al. 2021)).

Secondly, the AI generated descriptions are not fool proof. Thus, in order to
ensure the users can have a reasonably good experience, post-processing of AI
generated descriptions in the production platform is necessary to filter out
any inconsistent or low quality contents. Currently, this step is often based
on rules and involves humans in the loop. Nevertheless, it is a necessary
consideration during deployment to uphold user experience.

## Conclusions and Future Work

In this paper, we report our work on developing and deploying an automatic
product copywriting generation system (APCG) for the JD e-commerce product
recommendation platform. The proposed system consists of a transformer-pointer
network and a pre-trained sequence-to-sequence model to perform natural
language generation tasks in order to generate descriptions for products
automatically. The system incorporates automatic evaluation and human
screening modules to guard against the occasional low quality generated
textual contents and ensure a high level of user experience. By adopting the
APCG system, a high volume of 2.53 million product descriptions have been
generated over a seven month period. The overall average CTR achieved by the
JD Discover Goods Channel product recommendation platform is improved by
4.22%, while its CVR is improved by 3.61% on an year-on-year basis. The
accumulated GMV made by our system is improved by 213.42%, compared to the
start number in Feb 2021.

In future work, we will focus on enhancing the model performance and expanding
the application scenarios:

  * •

Domain-specific pre-trained models: our current experience shows that public
pre-trained models can help enhance the quality of natural language
generation. We will develop pre-trained models specific for the e-commerce
domain to further improve model performance.

  * •

Expending application scenarios: besides product description generation,
automatic natural language generation can benefit other application scenarios
as well. We will explore applying APCG to other application scenarios such as
personalized product description generation and multi-product advertisement
post generation.

## Acknowledgments

Han Yu is supported by the National Research Foundation, Singapore under its
AI Singapore Programme (AISG Award No: AISG2-RP-2020-019); the Nanyang
Assistant Professorship (NAP); and the RIE 2020 Advanced Manufacturing and
Engineering (AME) Programmatic Fund (No. A20G8b0102), Singapore. Any opinions,
findings and conclusions or recommendations expressed in this material are
those of the author(s) and do not reflect the views of National Research
Foundation, Singapore.

## References

  * Bahdanau, Cho, and Bengio (2016) Bahdanau, D.; Cho, K.; and Bengio, Y. 2016.  Neural Machine Translation by Jointly Learning to Align and Translate.  arXiv:1409.0473.
  * Banko, Mittal, and Witbrock (2000) Banko, M.; Mittal, V. O.; and Witbrock, M. J. 2000.  Headline generation based on statistical translation.  _ACL_ , 318–325.
  * Bowman et al. (2015) Bowman, S.; Angeli, G.; Potts, C.; and Manning, C. D. 2015.  A large annotated corpus for learning natural language inference.  In _EMNLP_ , 632–642.
  * Brown et al. (2020) Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Chen et al. (2019) Chen, Q.; Lin, J.; Zhang, Y.; Yang, H.; Zhou, J.; and Tang, J. 2019.  Towards knowledge-based personalized product description generation in e-commerce.  In _KDD_ , 3040–3050.
  * Chen et al. (2021) Chen, Y.; Li, B.; Yu, H.; Wu, P.; and Miao, C. 2021.  HyDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks.  In _AAAI_ , 7081–7089.
  * Devlin et al. (2019a) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019a.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _NAACL_ , 4171–4186.
  * Devlin et al. (2019b) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019b.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _ACL_.
  * Dong et al. (2019) Dong, L.; Yang, N.; Wang, W.; Wei, F.; Liu, X.; Wang, Y.; Gao, J.; Zhou, M.; and Hon, H.-W. 2019.  Unified Language Model Pre-training for Natural Language Understanding and Generation.  arXiv:1905.03197.
  * Gehring et al. (2017) Gehring, J.; Auli, M.; Grangier, D.; Yarats, D.; and Dauphin, Y. N. 2017.  Convolutional Sequence to Sequence Learning.  arXiv:1705.03122.
  * Keskar et al. (2019) Keskar, N. S.; McCann, B.; Varshney, L. R.; Xiong, C.; and Socher, R. 2019.  CTRL: A Conditional Transformer Language Model for Controllable Generation.  arXiv:1909.05858.
  * Knight and Marcu (2000) Knight, K.; and Marcu, D. 2000.  Statistics-based summarization-step one: Sentence compression.  _AAAI_ , 703–710.
  * Lafferty, McCallum, and Pereira (2001) Lafferty, J. D.; McCallum, A.; and Pereira, F. C. N. 2001.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data.  In _ICML_ , 282–289. San Francisco, CA, USA.
  * Lavie, Sagae, and Jayaraman (2004) Lavie, A.; Sagae, K.; and Jayaraman, S. 2004.  The significance of recall in automatic metrics for MT evaluation.  In _AMTA_ , 134–143.
  * Lewis et al. (2019) Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Li et al. (2020) Li, H.; Yuan, P.; Xu, S.; Wu, Y.; He, X.; and Zhou, B. 2020.  Aspect-aware multimodal summarization for chinese e-commerce products.  In _AAAI_ , volume 34, 8188–8195.
  * Li et al. (2018) Li, J.; Song, Y.; Zhang, H.; Chen, D.; Shi, S.; Zhao, D.; and Yan, R. 2018.  Generating Classical Chinese Poems via Conditional Variational Autoencoder and Adversarial Training.  In _EMNLP_ , 3890–3900.
  * Lin (2004) Lin, C.-Y. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , 74–81.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  Roberta: A robustly optimized bert pretraining approach.  In _ACL_.
  * Och et al. (2004) Och, F. J.; Gildea, D.; Khudanpur, S.; Sarkar, A.; Yamada, K.; Fraser, A.; Kumar, S.; Shen, L.; Smith, D.; Eng, K.; Jain, V.; Jin, Z.; and Radev, D. 2004\.  A Smorgasbord of Features for Statistical Machine Translation.  In _NAACL-HLT_ , 161–168.
  * Papineni et al. (2002) Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _ACL_ , 311–318.
  * Post (2018) Post, M. 2018.  A call for clarity in reporting BLEU scores.  _arXiv preprint arXiv:1804.08771_.
  * Radford et al. (2018a) Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I. 2018a.  Improving language understanding by generative pre-training.  _URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf_.
  * Radford et al. (2018b) Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2018b.  Language Models are Unsupervised Multitask Learners.
  * Radford et al. (2019) Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8).
  * Rajpurkar et al. (2016) Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016.  SQuAD: 100,000+ Questions for Machine Comprehension of Text.  In _EMNLP_ , 2383–2392.
  * Reiter and Dale (2000) Reiter, E.; and Dale, R. 2000.  _Building Natural Language Generation Systems_.  Studies in Natural Language Processing. Cambridge University Press.
  * Rush, Chopra, and Weston (2015) Rush, A. M.; Chopra, S.; and Weston, J. 2015.  A neural attention model for abstractive sentence summarization.  _EMNLP_ , 379–389.
  * Song et al. (2019) Song, K.; Tan, X.; Qin, T.; Lu, J.; and Liu, T.-Y. 2019.  MASS: Masked Sequence to Sequence Pre-training for Language Generation.  In _ICML_ , 5926–5936. PMLR.
  * Su (2021) Su, J. 2021.  T5 PEGASUS - ZhuiyiAI.  Technical report.
  * Sutskever, Vinyals, and Le (2014) Sutskever, I.; Vinyals, O.; and Le, Q. V. 2014.  Sequence to Sequence Learning with Neural Networks.  arXiv:1409.3215.
  * Tao et al. (2018) Tao, C.; Gao, S.; Shang, M.; Wu, W.; Zhao, D.; and Yan, R. 2018.  Get The Point of My Utterance! Learning Towards Effective Responses with Multi-Head Attention Mechanism.  In _IJCAI_ , 4418–4424.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention Is All You Need.  arXiv:1706.03762.
  * Wang et al. (2017) Wang, J.; Hou, Y.; Liu, J.; Cao, Y.; and Lin, C.-Y. 2017.  A statistical framework for product description generation.  In _IJCNLP_ , 187–192.
  * Wu et al. (2020) Wu, Z.; Pan, S.; Chen, F.; Long, G.; Zhang, C.; and Philip, S. Y. 2020.  A comprehensive survey on graph neural networks.  _IEEE Transactions on Neural Networks and Learning Systems_ , 32(1): 4–24.
  * Yao et al. (2019) Yao, L.; Peng, N.; Weischedel, R.; Knight, K.; Zhao, D.; and Yan, R. 2019.  Plan-and-Write: Towards Better Automatic Storytelling.  _AAAI_ , 33(01): 7378–7385.
  * Zhan et al. (2021) Zhan, H.; Zhang, H.; Chen, H.; Shen, L.; Ding, Z.; Bao, Y.; Yan, W.; and Lan, Y. 2021.  Probing Product Description Generation via Posterior Distillation.  In _AAAI_ , volume 35, 14301–14309.
  * Zhang et al. (2021) Zhang, X.; Jiang, Y.; Shang, Y.; Cheng, Z.; Zhang, C.; Fan, X.; Xiao, Y.; and Long, B. 2021.  DSGPT: Domain-Specific Generative Pre-Training of Transformers for Text Generation in E-commerce Title and Review Summarization.  In _SIGIR_ , 2146–2150.
  * Zhang, Zhao, and LeCun (2015) Zhang, X.; Zhao, J.; and LeCun, Y. 2015.  Character-level convolutional networks for text classification.  _NeurIPS_ , 28: 649–657.
  * Zhang et al. (2020) Zhang, Y.; Sun, S.; Galley, M.; Chen, Y.-C.; Brockett, C.; Gao, X.; Gao, J.; Liu, J.; and Dolan, W. B. 2020.  DIALOGPT: Large-Scale Generative Pre-training for Conversational Response Generation.  In _ACL_ , 270–278.
  * Zou et al. (2020) Zou, Y.; Zhang, X.; Lu, W.; Wei, F.; and Zhou, M. 2020.  Pre-training for Abstractive Document Summarization by Reinstating Source Text.  In _EMNLP_ , 3646–3660.


lands in a leaf containing no training points. (13) shows that the probability
of this event decays exponentially in $n$.


# MambaTalk: Efficient Holistic Gesture Synthesis
with Selective State Space Models

Zunnan Xu11footnotemark: 1 Yukang Lin11footnotemark: 1 Haonan
Han11footnotemark: 1 Sicheng Yang Ronghui Li Yachao Zhang22footnotemark: 2 Xiu
Li22footnotemark: 2
Tsinghua Shenzhen International Graduate School, Tsinghua University

###### Abstract

Gesture synthesis is a vital realm of human-computer interaction, with wide-
ranging applications across various fields like film, robotics, and virtual
reality. Recent advancements have utilized the diffusion model and attention
mechanisms to improve gesture synthesis. However, due to the high
computational complexity of these techniques, generating long and diverse
sequences with low latency remains a challenge. We explore the potential of
state space models (SSMs) to address the challenge, implementing a two-stage
modeling strategy with discrete motion priors to enhance the quality of
gestures. Leveraging the foundational Mamba block, we introduce MambaTalk,
enhancing gesture diversity and rhythm through multimodal integration.
Extensive experiments demonstrate that our method matches or exceeds the
performance of state-of-the-art models.

**footnotetext: Equal contribution††footnotetext: Corresponding author

##  1 Introduction

Gesture synthesis is a critical area of research in human-computer interaction
(HCI), which has very broad application prospects, such as film, robotics,
virtual reality, and digital human development [21]. The task is challenging
due to the variable correlation between speech and gestures, as the same
spoken content can elicit markedly different gestures among speakers.
Meanwhile, the generated gestures should synchronize with the speaker’s
rhythm, emotional cues, and intentions [26, 1, 6, 34].

Recent works in co-speech gesture generation have shown great progress [9, 38,
58, 57, 2, 52]. By introducing new datasets[62] and more modalities[61, 28],
previous work achieved end-to-end gesture generation based on RNN models. With
the success of transformer in nature language processing, recent works[60, 6,
35, 45] leverage the power of attention mechanism to generate more expressive
gestures that better synchronize with speech. Recently, with the development
of diffusion model in generative tasks, the latest works[63, 2, 53, 5, 56]
have applied the diffusion model to gesture synthesis, significantly improving
the diversity of generated gesture. However, the exploration of generation for
long holistic gesture sequences with low latency remains relatively uncharted,
constraining its application in dynamic, interactive environments. RNN-based
models often struggle with the long-term forgetting issue[44], which impairs
their ability to generate long sequences of gestures effectively.
Additionally, these models may produce gestures that lack variability, tending
towards an average representation [31]. Transformer-based models depend
heavily on subtle positional encoding to capture the order of input
elements[36, 41]. Meanwhile, their computational complexity, which grows
quadratically with the length of the input sequence, poses a challenge for
generating long sequences of gestures. For the diffusion-based model, the
intricate sampling strategy and iterative process lead to high computational
expenses, which hinder their broad adoption in gesture generation scenarios
that demand low latency.

Recently, state space models (SSMs) showcase enormous potential in tackling
long-range dependency challenges[10]. Inspired by continuous state space
models from control systems and enhanced by HiPPO initialization[11], SSMs
[14] show promise in addressing long-term forgetting issue. To mitigate the
computational and memory overhead of state representations, S4[13] proposes a
normalization technique that structures parameters into a diagonal form,
leading to the development of various structured SSMs with configurations such
as complex-diagonal structures[16, 12], MIMO support [40], diagonal-plus-low-
rank decomposition [18], and selection mechanisms [10]. These advancements
have been integrated into large-scale representation models [32, 33]. Some
pioneering works have applied SSMs for tasks like language understanding [32,
33], content-based reasoning [10], and visual recognition[30, 47]. In our
work, we further explore the potential of SSMs in holistic gesture synthesis.
We observe that directly applying the selective scan mechanism from Mamba[10]
to gesture generation would result in jittery outputs. To refine the generated
gestures, we implement a two-stage modeling strategy, which we augmented with
discrete motion priors derived from VQVAE[43]. This hybrid approach
effectively mitigates the jitter issue, enhancing the smoothness and overall
quality of the synthesized gestures. Considering the significant differences
in deformation and movement patterns among different body parts [51, 25], we
further propose using multiple Mamba blocks for refining the latent space
representations of various body segments. These approaches enable dynamic
interaction and iterative refinement of different body parts while maintaining
low latency, leading to more diverse and rhythmic gestures.

The main contributions of our work can be summarized as below:

  * •

We are the first to explore the potential of the selective scan mechanism for
gesture synthesis, integrating it with VQVAE to achieve low-latency holistic
gesture synthesis.

  * •

We present MambaTalk, an innovative framework for holistic gesture synthesis
that combines multiple Mamba blocks, each informed by distinct priors. The
combination enhances the quality of latent space representations, resulting in
a more refined adjustment of individual movement patterns throughout the body.

  * •

Extensive experiments and analyses demonstrate the effectiveness of the
proposed method.

##  2 Related Work

###  2.1 Co-speech Gesture Generation

Co-speech gesture generation aims to automatically generate gestures based on
speech input. Existing approaches can be broadly categorized into three
groups: (i) Rule-based methods: These methods rely on pre-defined rules and
gesture libraries to generate gestures based on speech features[20, 46]. While
offering interpretable results, they require significant manual effort in
creating gesture datasets and defining rules. (ii) Statistical models: These
approaches leverage data-driven techniques to learn mapping rules between
speech and gestures, often employing pre-defined gesture units[19, 22]. While
overcoming the limitations of manual rule creation, these methods still rely
on handcrafted features. (iii) Deep learning methods: Recent advancements in
deep learning have enabled neural networks to capture the complex relationship
between speech and gestures directly from raw multimodal data[61, 28, 59, 27].
This progress has established deep learning approaches, particularly recurrent
neural networks (RNNs)[61, 28, 51], transformers[3, 37], and diffusion
models[2, 63, 53], as the prevailing paradigm for co-speech gesture
generation.

However, each of these models suffers from certain limitations that hinder
their performance. RNNs inherently process sequences in a serial manner, where
each timestep’s computation depends on the output of the previous timestep.
This limits their ability to efficiently handle long sequences and introduces
cumulative latency. Meanwhile, RNNs lack inherent parallelism, further
restricting their potential for high-speed computation. Transformers consider
all positions within a sequence at every timestep, resulting in high
computational complexity, especially for long sequences. While diffusion
models significantly enhance the diversity of generated outputs, the sampling
process is computationally expensive, leading to significant latency in
generating outputs. To overcome these limitations, our method investigates the
capacity of selective state space models in the field of gesture synthesis. To
the best of our knowledge, we are the first to apply selective state space
models to the task of gesture generation.

###  2.2 Selective State Space Models

State Space Models (SSMs) are a novel class of models recently integrated into
deep learning for state space transformation[13, 8]. As foundational models
evolve, various subquadratic-time architectures have emerged, including linear
attention, gated convolution, recurrent models, and structured state space
models (SSMs), aimed at mitigating the computational inefficiencies of
Transformers when dealing with lengthy sequences. However, these advancements
have yet to match the performance of attention mechanisms in critical
modalities like language processing.

SSMs draw inspiration from continuous state space models in control systems
and, when combined with HiPPO initialization [11], as seen in LSSL [14], show
promise in tackling long-range dependency issues. However, the computational
and memory demands of the state representation render LSSL impractical for
real-world use. To address this, S4[13] suggests normalizing the parameters
into a diagonal structure. This has led to the emergence of various structured
SSMs with diverse configurations, such as complex-diagonal structures[16, 12],
multiple-input multiple-output (MIMO) support[40], diagonal-plus-low-rank
decomposition[18], and selection mechanisms[10]. These models have been
incorporated into large-scale representation models[32, 33].

These models primarily focus on the application of SSMs to long-range and
sequential data like language and speech, for tasks such as language
understanding[32, 33], content-based reasoning[10], and pixel-level 1-D image
classification[13]. Recently, some pioneering work[30, 47] have explored their
application in visual recognition. We further demonstrate that by
incorporating the selective scan mechanism from mamba[10] and the discrete
motion priors from VQVAE[43], our proposed MambaTalk is capable of matching
the performance of existing popular holistic gesture synthesis models like
EMAGE[27] and Talkshow[59], highlighting the potential of MambaTalk as a
powerful gesture synthesis model.

##  3 Method

We aim to synthesize sequential 3D holistic gestures from speech signals
(e.g., audio and text) using selective state space models. However, simply
applying such a model to gesture synthesis leads to severe gesture jittering
issues. We also found that it is difficult to maintain performance because one
speech could correspond to a variety of potential gestural expressions. To
overcome these challenges, we suggest initially characterizing the gesture
space using the acquired discrete motion patterns. Subsequently, we propose to
develop speech-conditioned selective state space models within this framework.
This approach is designed to enhance the model’s robustness against
uncertainties that arise from cross-modal discrepancies. Our framework
consists of two stages: (i) modeling the discrete gestures and facial motion
spaces (§3.1) and (ii) learning speech-conditioned selective state space
models (§3.2) to generate 3D holistic gestures.

###  3.1 Discrete Gestures and Facial Motion Spaces

To ensure visual realism in motion animations from speech signals, we learn
extra motion priors to depict accurate movements and natural expressions.
Building on this concept, we propose a method to represent the gesture motion
space using multiple discrete codebooks.

Motion Quantization. Considering the substantial variations in deformation
magnitude and periodicity among various body parts, our approach involves
learning multiple codebooks tailored for the reconstruction of distinct body
parts. For illustrative purposes, we detail the formulation of a single
codebook. For the codebook
$\mathcal{Z}=\left\\{\mathbf{z}_{k}\in\mathbb{R}^{C}\right\\}_{k=1}^{N}$, we
employ a set of allocated items $\\{\mathbf{z}_{k}\\}_{k\in\mathcal{S}}$ to
represent the holistic gesture motion $\mathbf{M}_{t}$. Here, $\mathcal{S}$
represents the chosen index sets. The element-wise quantization function
$Q(\cdot)$ maps each item $\hat{\mathbf{z}}_{t}$ in $\hat{\mathbf{Z}}$ to its
closest match $\mathbf{z}_{k}$ in the codebook $\mathcal{Z}$, as detailed by
Equation 1:

$\mathbf{Z}_{\mathbf{q}}=Q(\hat{\mathbf{Z}}):=\underset{\mathbf{z}_{k}\in\mathcal{Z}}{\arg\min}\left\|\hat{\mathbf{z}}_{t}-\mathbf{z}_{k}\right\|_{2},$
(1)

where the codebook entries act as the foundational motion elements within the
discrete motion space. To establish this, we follow[54, 27] to pre-train a
CNN-based Vector Quantized-Variational Autoencoder (VQ-VAE), which comprises
an encoder $E$, a decoder $D$, and a context-rich codebook $\mathcal{Z}$. This
is done through the self-reconstruction of holistic gesture motions. The
sequence of motions $\mathbf{M}_{1:T}$ is initially transformed into a
temporal feature representation $\hat{Z}=E(\mathbf{M}_{1:T})\in
R^{T^{\prime}\times H\times C}$, where $H$ represents the count of gesture
components, $T^{\prime}$ indicates the quantity of temporal units encoded
(with $P=\frac{T}{T^{\prime}}$ frames per unit). Subsequently, we derive the
quantized motion sequence $Z_{q}\in R^{T^{\prime}\times H\times C}$ by
quantization function $Q(\cdot)$. This function $Q$ maps each element in
$\hat{Z}$ to its closest corresponding entry within the codebook
$\mathcal{Z}$. The self-reconstruction is achieved as Equation 2:

$\mathbf{Z}_{\mathbf{q}}=Q\left(E\left(\mathbf{M}_{1:T}\right)\right),\hat{\mathbf{M}}_{1:T}=D\left(\mathbf{Z}_{\mathbf{q}}\right),$
(2)

The discrete gesture motion space minimizes mapping ambiguity with its finite
elements, while retaining expressiveness through its context-rich latent
representation.

Training objectives. For the training of the quantized autoencoder, we employ
motion-level losses to mitigate the jittering issue of generated gestures,
along with two intermediate losses at the code level:

$\displaystyle\mathcal{L}_{\mathrm{VQ}}=$
$\displaystyle\mathcal{L}_{rec}(\mathbf{M},\hat{\mathbf{M}})+\mathcal{L}_{vel}(\mathbf{M^{\prime}},\hat{\mathbf{M^{\prime}}})+\mathcal{L}_{acc}(\mathbf{M^{\prime\prime}},\hat{\mathbf{M^{\prime\prime}}})$
(3)
$\displaystyle+\left\|\operatorname{sg}(\hat{\mathbf{Z}})-\mathbf{Z}_{\mathbf{q}}\right\|_{2}^{2}+\left\|\hat{\mathbf{Z}}-\operatorname{sg}\left(\mathbf{Z}_{\mathbf{q}}\right)\right\|_{2}^{2},$

where $\mathbf{M^{\prime}}$ and $\mathbf{M^{\prime\prime}}$ means the velocity
and acceleration of motion, $sg(\cdot)$ denotes a stop-gradient operation,
$\mathcal{L}_{\text{rec }}$ are Geodesic[42] loss and the last two terms are
designed to refine the codebook entries. For facial motions, we utilize MSE
loss for both velocity ($\mathcal{L}_{\text{vel}}$) and acceleration
($\mathcal{L}_{\text{acc}}$) loss. For body motions, we use L1 loss as
$\mathcal{L}_{\text{vel}}$ and $\mathcal{L}_{\text{acc }}$. Additionally, for
the foot contact loss, we employ MSE loss as the loss function. These terms
work by minimizing the distance between the codebook $Z$ and the embedded
features $\hat{Z}$. Given that the quantization function (Equation 1) is non-
differentiable, we utilize the straight-through gradient estimator[43] to
propagate the gradients from the decoder input to the encoder output.

###  3.2 Speech-Driven Selective State Spaces Gesture Synthesis Model

Overall Framework. Utilizing the acquired discrete motion prior, we establish
a cross-modal mapping from speech inputs to target motion codes, enabling the
generation of realistic holistic gesture motions. In our approach to speech-
driven gesture synthesis, we utilize audio sequences
$A=\\{a_{1},\ldots,a_{N}\\}$ and text sequences $T=\\{t_{1},\ldots,t_{N}\\}$
as inputs to guide the generation of co-speech gestures
$G=\\{g_{1},\ldots,g_{N}\\}$. Here, $N$ signifies the total frame count, and
$g_{i}\in R^{55\times 6+100+4+3}$ denotes 55 pose joints in Rot6D, $R^{100}$
FLAME parameters, $R^{4}$ foot contact labels, $R^{3}$ global translations for
the $i$-th frame. The gesture synthesis model, comprising audio encoders
$E_{\text{A}}$ and text encoders $E_{\text{T}}$ and multiple selective state
space models $D_{\text{B}}$ for different parts of the body, is trained on the
discrete motion space, conditioned on the speech.

Speech Feature Extraction. For audio feature extraction, two convolutional-
based audio feature extraction networks are employed to extract corresponding
audio features for facial and body generations, respectively. Specifically, we
encode the amplitude and onset features of the raw audio to obtain audio
features $f_{A}=\\{fa_{1},...,fa_{N}\\}$. Similarly, for processing speech
input words, we follow [55, 28] to employ two pre-trained FastText [4] to
obtain word embeddings, which are then refined by linear projections to
produce text features $f_{T}=\\{ft_{1},...,ft_{N}\\}$. We further employ
multimodal fusion mechanisms [50, 48] to enhance features derived from speech,
thereby facilitating the learning process of the model.

Selective State Spaces Model for Motion Synthesis. In our approach, we adopt
the Mamba model, a structured state space sequence model that incorporates a
selection mechanism and a scan module (S6), for the task of gesture synthesis.
This model is particularly adept at sequence modeling, as it dynamically
selects salient input segments for prediction, thereby enhancing its focus on
pertinent information and improving overall performance. Unlike the
traditional S4 model, which uses time-invariant matrices $A$, $B$, $C$, and
scalar $\Delta$, Mamba introduces a novel selection mechanism that allows for
the learning of these parameters from the input data using fully-connected
layers. This adaptability enables Mamba to better generalize and perform
complex modeling tasks.

The Mamba model operates by defining the state space with structured matrices
that introduce specific constraints on the parameters, facilitating efficient
computation and data storage. For each batch and each dimension, the model
processes the input $x_{t}$, hidden state $h_{t}$, and output $y_{t}$ at each
time step $t$. The model’s formulation is as follows:

$\begin{gathered}h_{t}=\bar{A}_{t}h_{t-1}+\bar{B}_{t}x_{t},\\\
y_{t}=C_{t}h_{t},\end{gathered}$ (4)

where $\bar{A}_{t}$, $\bar{B}_{t}$, and $C_{t}$ are matrices and vectors that
are updated at each time step, allowing the model to adapt to the temporal
dynamics of the input sequence. With discretization, let $\Delta$ denote the
sampling interval, then

$\begin{gathered}\bar{A}=\exp(\Delta A),\\\ \bar{B}=(\Delta
A)^{-1}(\exp(\Delta A)-I)\cdot\Delta B,\\\
h_{t}=\bar{A}h_{t-1}+\bar{B}x_{t},\end{gathered}$ (5)

where $(\Delta A)^{-1}$ denotes the inverse of matrix $\Delta A$, $I$ denotes
the identity matrix. The scan module within Mamba is designed to capture
temporal patterns and dependencies across multiple time steps by applying a
set of trainable parameters or operations to each segment of the input
sequence. These operations are learned during training and help the model to
effectively process sequential data.

In the context of gesture synthesis, we utilize the Mamba model’s capabilities
by feeding it fused multi-modal features from previous modules. With a state
space dimension of $N=16$, the model is tasked with capturing temporal
dependencies and extracting relevant information to synthesize realistic and
coherent gestures.

Recognizing the diverse deformations and motion patterns in various body
parts, we use four specialized Mamba models, each with unique parameters, to
capture the nuanced latent representations of facial, upper, lower, and hand
motions. The extracted latent features are then fed into their respective VQ-
Decoders to produce the final motion predictions. Furthermore, we incorporate
a pretrained Global Motion Predictor $\tilde{t}=G(\tilde{g}_{l})$ to adeptly
tackle the issue of foot sliding.

Training Objectives. The model’s training objectives are succinctly defined by
a unified loss function that balances reconstruction and cross-entropy losses.
This function is crafted to optimize the generation of movements across four
key body motions: the face, upper body, lower body, and hands.

The latent reconstruction loss, represented by $L_{reclatent}$, is quantified
using the Mean Squared Loss (MSELoss). Here, $z_{i}$ corresponds to the true
latent vectors, while $\hat{z}_{i}$ are the vectors reconstructed by the
model. The latent reconstruction loss is expressed as:

$L_{reclatent}=\frac{1}{N}\sum_{i=1}^{N}\|z_{i}-\hat{z}_{i}\|^{2},$ (6)

where $N$ denotes the number of frames. Concurrently, to encourage diversity
in the generated motions, we optimize the cross-entropy loss for latent code
class classification $L_{cls}$. Specifically, we employ the Negative Log
Likelihood Loss (NLLLoss), where $y_{i}$ represents the true class labels for
each sample, and $\hat{y}_{i}$ denotes the model’s predicted class labels.
This loss is calculated as the negative sum of the logarithm of the predicted
probabilities for the correct classes, encapsulated in the formula:

$L_{cls}=-\frac{1}{N}\sum_{i=1}^{N}\sum_{c=1}^{C}y_{ic}\log(\hat{y}_{ic}),$
(7)

where $N$ signifies the total number of frames, $C$ is the total number of
classes, and $y_{ic}$ is a binary indicator of whether class $c$ is the
correct label for sample $i$.

The total loss $L$ is a weighted sum of the categorical and latent
reconstruction losses, with $\alpha$ and $\beta$ serving as hyper-parameters
to balance their contributions:

$L=\alpha L_{cls}+\beta L_{reclatent},$ (8)

where $\alpha=1$ and $\beta=3$ for hands, upper and lower body motion. For
facial motion, we set $\alpha=0$ and $\beta=3$. By optimizing the total loss,
the model is trained to generate diverse gesture results.

##  4 Experiments

###  4.1 Experiments Setting

We train and evaluate on the BEATX-standard dataset proposed by[27]. BEATX-
standard contains 60 hours of data with high finger quality for 25 speakers
(12 female and 13 male). The dataset comprises 1762 sequences, each with an
average duration of 65.66 seconds. Each sequence includes a continuous
response to a daily inquiry. We split BEATX-standard into 85%/7.5%/7.5% for
the train/val/test set, with the ratio kept the same for each speaker.

###  4.2 Implementation Details

We utilize the Adam optimizer with a learning rate of $2.5\times 10^{-4}$. To
maintain training stability, we apply gradient norm clipping at a value of
0.99. For the construction of the VQ-VAE, we adhere to the methodology
outlined in [15]. Specifically, the codebook entries are set to feature
lengths of 512, and the codebook size is established at 256. The VQ-VAE is
trained for 200 epochs, with a learning rate of $2.5\times 10^{-4}$ for the
first 195 epochs, which is then reduced to $2.5\times 10^{-5}$ for the final 5
epochs. During the second stage, the model is trained for 100 epochs. All
experiments are conducted using one NVIDIA A100 GPU.

###  4.3 Evaluation Metrics

To evaluate the realism of body gestures, we employ Fréchet Gesture Distance
(FGD)[61] to measure the proximity of the distribution between the ground
truth and generated body gestures. Subsequently, Diversity[23] is quantified
by computing the average L1 distance across multiple body gesture clips. The
synchronization between speech and motion is achieved using Beat Constancy
(BC)[24]. For facial motions, we assess positional accuracy by calculating the
vertex Mean Squared Error (MSE)[49]. Additionally, the difference between the
ground truth and the generated facial vertices is measured using the vertex L1
difference (LVD)[59].

###  4.4 Qualitative Analysis

User Study. We conducted a user study to assess the visual quality of the
generated co-speech 3D gestures. For each method under comparison, we produced
10 gesture samples, which were then converted into video clips for evaluation
by 18 participants.

In each evaluation session, participants were presented with 20-second video
clips generated by various models. Instead of directly assigning scores to the
four metrics, they were instructed to assess the clips by comparing them in
pairs [39] across the following dimensions: (i) naturalness, (ii)
appropriateness, and (iii) synchrony. For naturalness, they evaluated the
similarity of the generated gestures to those made by humans, paying attention
to the authenticity and smoothness of the movements. In terms of
appropriateness, they considered the alignment of the gestures with the spoken
content, taking into account both the explicit meaning and the underlying
semantics. For synchrony assessment, they examined the timing of the gestures
in relation to the speech rhythm, audio, and facial expressions to ensure a
harmonious and integrated performance. They were asked to identify which clip
performed better in each comparison.

Table 1: User study results on BEATX. In the pairwise comparison analysis, the
win percentages represent the frequency with which each method outperformed
its alternative. The final column consolidates these percentages, presenting
the average win rate (%) for each method across all evaluations.

Method | CaMN[28] | EMAGE[27] | Ours | Average
---|---|---|---|---
CaMN Wins | - | 14.44% | 5.00% | 9.72%
EMAGE Wins | 85.56% | - | 28.89% | 57.22%
Ours Wins | 95.00% | 71.11% | - | 83.06%

We compared three methods: CaMN [28], EMAGE [27], our proposed method, and the
ground truth. As presented in Table 1, our method’s average scores are higher
than previous methods in the pairwise comparison. Our approach demonstrates
superiority not only in individual pairwise comparisons but also exhibits a
significant advantage in the average outcomes.

Visualizations. In Figure 1, we present the facial motion results generated by
our method, showcasing the generation of facial expressions and movements with
a high level of realism.

Figure 1: Visualization of the facial motions generated by CaMN, EMAGE and our
method. Unreasonable results are indicated by red and gray boxes and
reasonable ones by green boxes.

Our method effectively synchronizes with the phonetic articulation of speech
content, accurately reflecting the physical demands of pronunciation. For
instance, when uttering “walking”, “came” or “bus”, our approach ensures that
the mouth’s movements, such as opening, correspond closely with the actual
phonetic requirements. Other methods do not consistently achieve this level of
accuracy in aligning with the phonetic and physical nuances of speech. Our
method adeptly handles the subtleties of mouth closure and elongation required
for sounds such as “in”, closely aligning with the ground truth, whereas other
approaches may exhibit inconsistencies in this regard. Moreover, in instances
of silence, all methods, including ours, demonstrate a good capacity to learn
and maintain the mouth’s closed position, effectively reflecting the
underlying patterns of speech and silence. Since CaMN does not specifically
target the generation of facial movements, it results in a lack of variation
in facial expressions throughout the process.

As illustrated in Figure 2, our methodology generates gestures that
demonstrate improved rhythmic synchronization and a more lifelike appearance,
effectively capturing the essence of the speaker’s rhythmic patterns. For
example, in the expression “actually” our method guides the individual to
bring the hands inward in front of the chest, a subtle gesture not observed in
the results produced by CaMN and EMAGE, where the arms are either hang limply
at the sides or are splayed downward. Furthermore, in the depiction of “on the
way back” our approach accurately reflects the ground truth by slightly
bending down and raising one hand, while EMAGE cannot respond accurately to
this and remains standing.

Figure 2: Visualization of the gestures generated by CaMN, EMAGE and our
method. Unreasonable results are indicated by red boxes and reasonable ones by
green boxes.

Furthermore, our approach accurately captures the semantic essence of
movements. For instance, in response to the cue “hug” our method generates an
inward-circling motion of the arms, aligning perfectly with the ground truth,
which is a nuanced semantic element that other methodologies neglect.
Similarly, in scenarios such as “so small”, the result of our method is
similar to the ground truth, with the character’s hand moving inward. This
attention to detail ensures semantic consistency, which is lacking in other
approaches where actions are not aligned with the intended meaning.

As depicted in Figure 3, our approach yields gestures that exhibit enhanced
rhythmic alignment and a more natural appearance, closely mirroring the
speaker’s cadence. For instance, when conveying “we were”, our method
instructs the subject to hold both hands in front of the chest, a nuanced
detail absent in both CaMN and EMAGE’s outcomes, where either one or both arms
hang down. Additionally, when representing “no place to”, our method aligns
with the ground truth by extending both arms upwards, whereas CaMN and EMAGE
have their arms tucked in next to the body. Furthermore, our method accurately
captures the semantics of movement. In the case of “up”, our generated result
raises the right arm in alignment with the ground truth, a semantic detail
overlooked by other methods. Even in the context of “moving around” where our
left and right arm swings may differ from the ground truth, the overall
movement remains consistent.

Figure 3: Visualization of the gestures generated by CaMN, EMAGE and our
method. Unreasonable results are indicated by red boxes and reasonable ones by
green boxes.

Interestingly, for “sound of gunfire”, a difficult semantic for the model to
learn, our method still generates the result of the character’s right hand
clenched in a fist and the arm bent to indicate a tense situation. Notably,
for the emotion of fear expressed by “is horrible”, the result of our method
is similar to the ground truth, with the character’s hands hanging down and
his face facing downward, which is a very good visual representation of the
psychological state of panic and fear. In addition, as illustrated in Figure
3, our generated motions exhibit not only diverse characteristics, such as the
range of motion and which hands to use, but also a high degree of consistency
with the ground truth.

To sum up, our method ensures the creation of expressive gestures that are not
only natural-looking but also in sync with the rhythm of speech.

###  4.5 Quantitative Results

As shown in Table 2, our method attains the highest BC, with a significant
increase of 13.83% when compared to the previously top-performing method,
which underscores the superior capability of MambaTalk in discerning and
correlating audio-motion beats. Furthermore, our method performs competitively
with EMAGE in terms of FGD, and much better than others. This emphasizes the
high quality and naturalness of our generated movements, showing the ability
of MambaTalk to capture real motion dynamics. This also demonstrates the
authenticity of our generated motions, affirming the successful capture of
inherent motion characteristics.

Table 2: Quantitative results on BEATX. FGD (Frechet Distance) multiplied by $10^{-1}$, BC (Beat Constancy) multiplied by $10^{-1}$, Diversity, MSE (Mean Squared Error) multiplied by $10^{-7}$, and LVD (Learned Vector Distance) multiplied by $10^{-5}$. The best outcomes are in bold, and the next best are underlined.  Method | FGD $\downarrow$ | BC $\uparrow$ | Diversity $\uparrow$ | MSE $\downarrow$ | LVD $\downarrow$
---|---|---|---|---|---
FaceFormer[7] | - | - | - | 1.195 | 8.824
CodeTalker[49] | - | - | - | 1.243 | 8.877
S2G[9] | 25.129 | $\underline{6.902}$ | 7.783 | - | -
Trimodal[61] | 19.759 | 6.442 | 8.894 | - | -
HA2G[29] | 19.364 | 6.601 | 9.671 | - | -
DisCo[26] | 21.170 | 6.571 | 10.378 | - | -
CaMN[28] | 8.752 | 6.731 | 9.279 | - | -
DiffStyleGesture[53] | 10.137 | 6.891 | 11.075 | - | -
Habibie et al[17] | 14.581 | 6.779 | 8.874 | 1.445 | 9.548
TalkShow[59] | 7.313 | 6.783 | $\underline{12.859}$ | 1.399 | 9.488
EMAGE[27] | $\mathbf{5.423}$ | 6.794 | $\mathbf{13.057}$ | $\underline{1.180}$ | $\underline{8.715}$
MambaTalk (Ours) | $\underline{5.951}$ | $\mathbf{8.010}$ | 12.401 | $\mathbf{0.760}$ | $\mathbf{7.531}$

As for Diversity, our method performs on par with EMAGE and TalkShow, which
reflects our method’s prowess in generating a diverse set of movements. This
capability significantly enriches the overall expressiveness and variety of
the synthesized gestures. Moreover, our method outperforms its counterparts in
terms of MSE and LVD, with substantial improvements of 36% and 16%,
respectively. These two enhancements highlight the superior accuracy and
fidelity of our method in capturing fine-grained details, affirming its
efficacy in synthesizing realistic and authentic gestures.

###  4.6 Training Speed

Training efficiency is a critical factor in practical applications. We
compared MambaTalk’s training speed with existing methods.

Table 3: Comparison of training time with existing methods, where the
percentages denote the ratio of the training time required by other methods
relative to our method.

Method | Time per Epoch (s) | Epochs
---|---|---
CaMN[28] | 493(1173%) | 120
EMAGE[27] | 83(212%) | 400
MambaTalk (Ours) | 42(100%) | 100

In our evaluation of training speed, we utilized the speaker2 subset from the
BEATX dataset. When it comes to epoch time, we specifically calculate the
duration needed to train each individual epoch. As for the total time, the
reported figure also includes the time spent on validation and testing phases.
As demonstrated in Table 3, MambaTalk outperforms existing methods in terms of
training efficiency. The training time for a single epoch of MambaTalk is only
42 seconds. Comparatively, CaMN and EMAGE require 493 seconds and 83 seconds
per epoch, respectively, which translates to 1173% and 212% of MambaTalk’s
epoch time. When considering the total time, MambaTalk also boasts a faster
convergence rate, with our method requiring only 100 epochs. This represents a
significant reduction in training time, enabling faster model development and
iteration, which is especially advantageous in research and industry settings
where time is crucial.

The hardware acceleration design of the selective scan mechanism is a
testament to the potential of utilizing modern computational resources to
improve efficiency. By streamlining the training process and reducing the
computational overhead, MambaTalk not only saves valuable time but also opens
up new possibilities for large-scale model training that were previously
hindered by lengthy training durations.

##  5 Limitations

Currently, our approach to gesture synthesis involves using distinct modules
to animate various body parts, which naturally introduces some latency.
Developing a single, unified model capable of capturing the wide-ranging and
intricate deformations and motion patterns characteristic of different body
parts can be addressed in future research. This enhancement is anticipated to
lower computational overhead and substantially reduce the processing time,
thereby improving the real-time capabilities of the pipeline and ensuring a
smoother and more responsive gesture generation system.

##  6 Conclusion

In this study, we propose a framework to employ the state space models in
gesture synthesis. To alleviate the problem of jitter in gesture synthesis, we
have implemented discrete action priors, which enhance the effectiveness of
the selective scan mechanism and lead to smoother results. To enhance the
refinement of motion features in the latent space, we further incorporate
multiple mamba blocks with hybrid multi-modal fusion strategy. These modules
capture the subtle movements and deformations of various body parts, thereby
enhancing the overall quality of the generated gestures. By utilizing a linear
time series modeling strategy with selective state space, our method achieves
high-quality full body gesture generation with low latency.

## References

  * [1] Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu.  Rhythmic gesticulator: Rhythm-aware co-speech gesture synthesis with hierarchical neural embeddings.  ACM Transactions on Graphics (TOG), 41(6):1–19, 2022.
  * [2] Tenglong Ao, Zeyi Zhang, and Libin Liu.  Gesturediffuclip: Gesture diffusion model with clip latents.  arXiv preprint arXiv:2303.14613, 2023.
  * [3] Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha.  Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents.  In 2021 IEEE virtual reality and 3D user interfaces (VR), pages 1–10. IEEE, 2021.
  * [4] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov.  Enriching word vectors with subword information.  Transactions of the association for computational linguistics, 5:135–146, 2017.
  * [5] Junming Chen, Yunfei Liu, Jianan Wang, Ailing Zeng, Yu Li, and Qifeng Chen.  Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expression and gesture generation.  arXiv preprint arXiv:2401.04747, 2024.
  * [6] Kiran Chhatre, Radek Daněček, Nikos Athanasiou, Giorgio Becherini, Christopher Peters, Michael J Black, and Timo Bolkart.  Emotional speech-driven 3d body animation via disentangled latent diffusion.  arXiv preprint arXiv:2312.04466, 2023.
  * [7] Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura.  Faceformer: Speech-driven 3d facial animation with transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18770–18780, 2022.
  * [8] Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré.  Hungry hungry hippos: Towards language modeling with state space models.  arXiv preprint arXiv:2212.14052, 2022.
  * [9] Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik.  Learning individual styles of conversational gesture.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3497–3506, 2019.
  * [10] Albert Gu and Tri Dao.  Mamba: Linear-time sequence modeling with selective state spaces.  arXiv preprint arXiv:2312.00752, 2023.
  * [11] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré.  Hippo: Recurrent memory with optimal polynomial projections.  Advances in neural information processing systems, 33:1474–1487, 2020.
  * [12] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré.  On the parameterization and initialization of diagonal state space models.  Advances in Neural Information Processing Systems, 35:35971–35983, 2022.
  * [13] Albert Gu, Karan Goel, and Christopher Re.  Efficiently modeling long sequences with structured state spaces.  In International Conference on Learning Representations, 2021.
  * [14] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré.  Combining recurrent, convolutional, and continuous-time models with linear state space layers.  Advances in neural information processing systems, 34:572–585, 2021.
  * [15] Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng.  Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts.  In European Conference on Computer Vision, pages 580–597. Springer, 2022.
  * [16] Ankit Gupta, Albert Gu, and Jonathan Berant.  Diagonal state spaces are as effective as structured state spaces.  Advances in Neural Information Processing Systems, 35:22982–22994, 2022.
  * [17] Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed Elgharib, and Christian Theobalt.  Learning speech-driven 3d conversational gestures from video.  In Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents, pages 101–108, 2021.
  * [18] Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus.  Liquid structural state-space models.  In The Eleventh International Conference on Learning Representations, 2022.
  * [19] Michael Kipp, Michael Neff, Kerstin H Kipp, and Irene Albrecht.  Towards natural gesture synthesis: Evaluating gesture units in a data-driven approach to gesture synthesis.  In Intelligent Virtual Agents: 7th International Conference, IVA 2007 Paris, France, September 17-19, 2007 Proceedings 7, pages 15–28. Springer, 2007.
  * [20] Stefan Kopp and Ipke Wachsmuth.  Synthesizing multimodal utterances for conversational agents.  Computer animation and virtual worlds, 15(1):39–52, 2004.
  * [21] Taras Kucherenko, Patrik Jonell, Youngwoo Yoon, Pieter Wolfert, and Gustav Eje Henter.  A large, crowdsourced evaluation of gesture generation systems on common data: The genea challenge 2020.  In 26th international conference on intelligent user interfaces, pages 11–21, 2021.
  * [22] Sergey Levine, Philipp Krähenbühl, Sebastian Thrun, and Vladlen Koltun.  Gesture controllers.  In ACM SIGGRAPH 2010 papers, pages 1–11, 2010.
  * [23] Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao.  Audio2gestures: Generating diverse gestures from speech audio with conditional variational autoencoders.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11293–11302, 2021.
  * [24] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa.  Ai choreographer: Music conditioned 3d dance generation with aist++.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13401–13412, 2021.
  * [25] Yukang Lin, Haonan Han, Chaoqun Gong, Zunnan Xu, Yachao Zhang, and Xiu Li.  Consistent123: One image to highly consistent 3d asset using case-aware diffusion priors.  arXiv preprint arXiv:2309.17261, 2023.
  * [26] Haiyang Liu, Naoya Iwamoto, Zihao Zhu, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng.  Disco: Disentangled implicit content and rhythm learning for diverse co-speech gestures synthesis.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 3764–3773, 2022.
  * [27] Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Naoya Iwamoto, Bo Zheng, and Michael J Black.  Emage: Towards unified holistic co-speech gesture generation via masked audio gesture modeling.  arXiv preprint arXiv:2401.00374, 2023.
  * [28] Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng.  Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII, pages 612–630. Springer, 2022.
  * [29] Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou.  Learning hierarchical cross-modal association for co-speech gesture generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10462–10472, 2022.
  * [30] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu.  Vmamba: Visual state space model.  arXiv preprint arXiv:2401.10166, 2024.
  * [31] Shuhong Lu, Youngwoo Yoon, and Andrew Feng.  Co-speech gesture synthesis using discrete gesture token learning.  arXiv preprint arXiv:2303.12822, 2023.
  * [32] Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer.  Mega: Moving average equipped gated attention.  In The Eleventh International Conference on Learning Representations, 2022.
  * [33] Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur.  Long range language modeling via gated state spaces.  In International Conference on Learning Representations, 2023.
  * [34] Simbarashe Nyatsanga, Taras Kucherenko, Chaitanya Ahuja, Gustav Eje Henter, and Michael Neff.  A comprehensive review of data-driven co-speech gesture generation.  In Computer Graphics Forum, volume 42, pages 569–596. Wiley Online Library, 2023.
  * [35] Kunkun Pang, Dafei Qin, Yingruo Fan, Julian Habekost, Takaaki Shiratori, Junichi Yamagishi, and Taku Komura.  Bodyformer: Semantics-guided 3d body gesture synthesis with transformer.  ACM Transactions on Graphics (TOG), 42(4):1–12, 2023.
  * [36] Ofir Press, Noah A Smith, and Mike Lewis.  Train short, test long: Attention with linear biases enables input length extrapolation.  arXiv preprint arXiv:2108.12409, 2021.
  * [37] Xingqun Qi, Chen Liu, Lincheng Li, Jie Hou, Haoran Xin, and Xin Yu.  Emotiongesture: Audio-driven diverse emotional co-speech 3d gesture generation.  arXiv preprint arXiv:2305.18891, 2023.
  * [38] Shenhan Qian, Zhi Tu, Yihao Zhi, Wen Liu, and Shenghua Gao.  Speech drives templates: Co-speech gesture synthesis with learned templates.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11077–11086, 2021.
  * [39] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu.  Bailando: 3d dance generation by actor-critic gpt with choreographic memory.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11050–11059, 2022.
  * [40] Jimmy TH Smith, Andrew Warrington, and Scott Linderman.  Simplified state space layers for sequence modeling.  In The Eleventh International Conference on Learning Representations, 2022.
  * [41] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu.  Roformer: Enhanced transformer with rotary position embedding.  Neurocomputing, 568:127063, 2024.
  * [42] Tommi Tykkälä, Cédric Audras, and Andrew I Comport.  Direct iterative closest point for real-time visual odometry.  In 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pages 2050–2056. IEEE, 2011.
  * [43] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  Advances in neural information processing systems, 30, 2017.
  * [44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [45] Hendric Voß and Stefan Kopp.  Aq-gt: a temporally aligned and quantized gru-transformer for co-speech gesture synthesis.  arXiv preprint arXiv:2305.01241, 2023.
  * [46] Petra Wagner, Zofia Malisz, and Stefan Kopp.  Gesture and speech in interaction: An overview, 2014.
  * [47] Ziyang Wang, Jian-Qing Zheng, Yichi Zhang, Ge Cui, and Lei Li.  Mamba-unet: Unet-like pure visual mamba for medical image segmentation.  arXiv preprint arXiv:2402.05079, 2024.
  * [48] Yicheng Xiao, Zhuoyan Luo, Yong Liu, Yue Ma, Hengwei Bian, Yatai Ji, Yujiu Yang, and Xiu Li.  Bridging the gap: A unified video comprehension framework for moment retrieval and highlight detection.  arXiv preprint arXiv:2311.16464, 2023.
  * [49] Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong.  Codetalker: Speech-driven 3d facial animation with discrete motion prior.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12780–12790, 2023.
  * [50] Zunnan Xu, Zhihong Chen, Yong Zhang, Yibing Song, Xiang Wan, and Guanbin Li.  Bridging vision and language encoders: Parameter-efficient tuning for referring image segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17503–17512, 2023.
  * [51] Zunnan Xu, Yachao Zhang, Sicheng Yang, Ronghui Li, and Xiu Li.  Chain of generation: Multi-modal gesture synthesis via cascaded conditional control.  arXiv preprint arXiv:2312.15900, 2023.
  * [52] Sicheng Yang, Zilin Wang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Qiaochu Huang, Lei Hao, Songcen Xu, Xiaofei Wu, Changpeng Yang, et al.  Unifiedgesture: A unified gesture synthesis model for multiple skeletons.  In Proceedings of the 31st ACM International Conference on Multimedia, pages 1033–1044, 2023.
  * [53] Sicheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, Ming Cheng, and Long Xiao.  Diffusestylegesture: Stylized audio-driven co-speech gesture generation with diffusion models.  arXiv preprint arXiv:2305.04919, 2023.
  * [54] Sicheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, and Haolin Zhuang.  Qpgesture: Quantization-based and phase-guided motion matching for natural speech-driven gesture generation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pages 2321–2330. IEEE, June 2023.
  * [55] Sicheng Yang, Zhiyong Wu, Minglei Li, Mengchen Zhao, Jiuxin Lin, Liyang Chen, and Weihong Bao.  The reprgesture entry to the genea challenge 2022.  In Proceedings of the 2022 International Conference on Multimodal Interaction, pages 758–763, 2022.
  * [56] Sicheng Yang, Zunnan Xu, Haiwei Xue, Yongkang Cheng, Shaoli Huang, Mingming Gong, and Zhiyong Wu.  Freetalker: Controllable speech and text-driven gesture generation based on diffusion models for enhanced speaker naturalness.  arXiv preprint arXiv:2401.03476, 2024.
  * [57] Sicheng Yang, Haiwei Xue, Zhensong Zhang, Minglei Li, Zhiyong Wu, Xiaofei Wu, Songcen Xu, and Zonghong Dai.  The diffusestylegesture+ entry to the genea challenge 2023.  In Proceedings of the 25th International Conference on Multimodal Interaction, pages 779–785, 2023.
  * [58] Payam Jome Yazdian, Mo Chen, and Angelica Lim.  Gesture2vec: Clustering gestures using representation learning methods for co-speech gesture generation.  In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3100–3107. IEEE, 2022.
  * [59] Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black.  Generating holistic 3d human motion from speech.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 469–480, 2023.
  * [60] Lianying Yin, Yijun Wang, Tianyu He, Jinming Liu, Wei Zhao, Bohan Li, Xin Jin, and Jianxin Lin.  Emog: Synthesizing emotive co-speech 3d gesture with diffusion model.  arXiv preprint arXiv:2306.11496, 2023.
  * [61] Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee.  Speech gesture generation from the trimodal context of text, audio, and speaker identity.  ACM Transactions on Graphics (TOG), 39(6):1–16, 2020.
  * [62] Youngwoo Yoon, Woo-Ri Ko, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee.  Robots learn social skills: End-to-end learning of co-speech gesture generation for humanoid robots.  In 2019 International Conference on Robotics and Automation (ICRA), pages 4303–4309. IEEE, 2019.
  * [63] Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, and Lequan Yu.  Taming diffusion models for audio-driven co-speech gesture generation.  arXiv preprint arXiv:2303.09119, 2023.