#  Fine-Tuning Large Language Models for Stock Return Prediction Using
Newsflow

Tian Guo Emmanuel Hauptmann
Systematic Equities Team
RAM Active Investments, Geneva, Switzerland  Correspondence to<EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs) and their fine-tuning techniques have
demonstrated superior performance in various language understanding and
generation tasks. This paper explores fine-tuning LLMs for stock return
forecasting with financial newsflow. In quantitative investing, return
forecasting is fundamental for subsequent tasks like stock picking, portfolio
optimization, etc. We formulate the model to include text representation and
forecasting modules. We propose to compare the encoder-only and decoder-only
LLMs, considering they generate text representations in distinct ways. The
impact of these different representations on forecasting performance remains
an open question. Meanwhile, we compare two simple methods of integrating
LLMs’ token-level representations into the forecasting module. The experiments
on real news and investment universes reveal that: (1) aggregated
representations from LLMs’ token-level embeddings generally produce return
predictions that enhance the performance of long-only and long-short
portfolios; (2) in the relatively large investment universe, the decoder LLMs-
based prediction model leads to stronger portfolios, whereas in the small
universes, there are no consistent winners. Among the three LLMs studied
(DeBERTa, Mistral, Llama), Mistral performs more robustly across different
universes; (3) return predictions derived from LLMs’ text representations are
a strong signal for portfolio construction, outperforming conventional
sentiment scores.

##  1 Introduction

Quantitative investing relies on extracting quantitative features or signals
from various data sources including market prices, economic indicators,
financial text, etc., to build and optimize investment portfolios [11, 2]. In
recent years, the use of text data for quantitative investing has grown
significantly, thanks to the advancement of natural language processing (NLP)
techniques [45, 35, 30]. In particular, large language models (LLMs) have
demonstrated superior performance on various language understanding and
generation tasks [14, 5, 19, 37], and the fine-tuning technique allows for
adapting the pre-trained LLMs to fit investing-related applications [16, 10].

This paper is focused on return forecasting with financial news for stock
picking. Return forecasting is useful for picking stocks with profit
potentials to include in portfolios. Financial news reports on events and
announcements related to companies, industries, the economy, etc., and shows
notable predictive power for stock future performance in previous studies [27,
17].

The conventional way of applying financial news data to stock picking involves
a multi-step extraction-and-validation process as illustrated in Fig. 1(a),
i.e., formulating the numerical features (e.g., sentiments, topics,
popularity, etc.) with the expectation that these features have a predictive
relationship with stock future performance (e.g., forward return, volatility,
etc.) [1, 36], developing the calculation processes or machine learning models
to extract features from the news (e.g., train a financial sentiment
classification model), and validating the predictive power of extracted
features by statistical analysis or building forecasting models. This process
might be time-consuming and require additional data (e.g., labeled financial
sentiment data) and continuous refinements.

LLMs generate numerical representations (or embeddings) of text that capture
semantic relations, and these representations can naturally serve as features
for forecasting tasks. Given this intuition, this paper explores direct news-
to-return prediction through fine-tuning LLMs. Fig. 1 illustrates the
difference between the conventional feature extraction-and-validation process
and our LLM-based news-to-return process. Though some previous works attempted
to use text embedding for forecasting [27, 41, 30, 13], few works have
explored the potential of fine-tuning LLMs for stock return forecasting with
newsflow. Moreover, this paper has the contribution as follows:

Figure 1:  Comparison of different workflows of utilizing financial news for
stock picking. (a) Conventional feature extraction-and-validation process,
e.g., financial sentiments. (b) News-to-return forecasting by fine-tuning
LLMs.

  * •

We design an LLM-based return prediction model comprising the text
representation and the forecasting modules.

  * •

We hypothesize that the text representations from encoder-only and decoder-
only LLMs will perform differently due to their distinct methods of encoding
text sequences during pre-training and fine-tuning; thus we propose to compare
the encoder-only (DeBERTa) and decoder-only LLMs (Mistral, Llama3) as the
representation module of the prediction model.

  * •

Considering that LLM-generated text representations are at the token level, we
present two simple methods to integrate token representations into the
forecasting module: bottleneck representations and aggregated representations.

  * •

We perform experiments on real financial news and various investment
universes. In addition to evaluating prediction errors, we assess two types of
portfolios built on return predictions through backtesting in out-of-sample
periods. The experimental comparison between encoder-only and decoder-only
LLMs, as well as between bottleneck and aggregated representations, offers
insights for identifying suitable text representations for different investing
strategies and markets.

##  2 Related Work

Numerous works have investigated using financial text data for forecasting
tasks. Previous works mostly used word-level embedding techniques lacking the
ability of contextual modeling. [44, 45] extracted the sentiment score from
financial newsflow, social media, and tweets for stock price predicting. [27,
17] explored learning numeric representations of financial news by attention
mechanisms for modeling stock movements. [41] studied combining sentiment and
text representations for return prediction. [7] studied the embedding
aggregate strategy of news for forex prediction.

The advent of LLMs and related techniques provides a new powerful way of using
text data for forecasting tasks in quantitative investing [47, 26]. LLMs can
be broadly categorized into three main types. Encoder-only models such as BERT
(Bidirectional Encoder Representations from Transformers) [9] and DeBERTa
(Decoding-enhanced BERT with disentangled attention) [15, 14], focus on
learning contextual embeddings for input text. Decoder-only models like GPT-3
(Generative Pre-trained Transformer 3) [31] and Mistral [19] are trained to
generate text by predicting the next token in a sequence. Encoder-decoder
models including T5 (Text-To-Text Transfer Transformer) [25] and BART
(Bidirectional and Auto-Regressive Transformers) [33] are a mix of both
encoder and decoder architectures and suitable for sequence-to-sequence tasks
such as machine translation, summarization, and question-answering.

LLMs are pre-trained on vast amounts of text data to learn general language
patterns. Following pre-training, there are two main approaches to applying
LLMs to downstream tasks. The prompt technique is to design specific inputs to
guide the pre-trained LLM to produce the desired output without modifying the
LLM’s parameters [32, 6, 21]. The second approach is to fine-tune LLMs by
adjusting the pre-trained LLM’s parameters to adapt to specific tasks [12, 42,
10, 8]. In particular, parameter-efficient fine-tuning techniques have gained
popularity [16, 10, 28]. For instance, LoRA (Low-Rank Adaptation) [16]
introduces low-rank adaptations to the pre-trained model parameters, thereby
reducing the computational and memory overhead of fine-tuning.

Some recent works use LLMs as feature extractors to obtain predictive signals
from text. Authors in [3, 29] explored the fine-tuning of pre-trained LLMs to
provide more accurate financial sentiment analysis. Instead of fine-tuning
LLMs, [40] extracted factors from the financial news and price history by
prompts on generative LLMs. [20] used chain-of-thought prompts [43] on
generative LLMs to analyze financial statements.

Unlike existing works that extract features from text using LLMs, this paper
focuses on fine-tuning LLMs to directly model the relation between financial
text and stocks’ future performance, i.e., newsflow and forward return.
Meanwhile, we evaluate the text representations from different types of LLMs
to study their different effectiveness for the return forecasting task.

##  3 From Financial Newsflow to Stock Portfolios through LLMs

###  3.1 Problem Statement

Assume an investment universe consisting of a set of stocks denoted by
$\mathcal{U}=\\{s\\}_{s=1}^{S}$, where $s$ represents the stock index. In
quantitative investing, the stock-picking process selects a subset of the
universe as the investing portfolio based on quantitative criteria. As market
conditions and various information change, the stock-picking process is
repeatedly performed to update or rebalance the portfolios at (regular) time
intervals, e.g., weekly, monthly, etc.

Figure 2:  Illustration of the LLM-based return forecasting model for the
stock-picking process. Assume an investment universe of 3 stocks denoted by
$a,b,c$. Each stock has an associated list of news. Then, given the return
forecasts and ranks, stocks can be selected into long-only or long-short
portfolios.

This paper is interested in predicting stock returns with news for stock
picking. Specifically, let $r_{s,t+n}\in\mathbb{R}$ be the $n$-step forward
return of stock $s$ w.r.t. timestep $t$. The textual content of financial news
reported at time $i$ and w.r.t. stock $s$ is denoted by $\mathbf{x}_{s,i}$, a
list of text tokens. At time $t$, the news text available for predicting
$r_{s,t+\ell}$ in a look-back time window $W$ is
$\\{\mathbf{x}_{s,i}\\}_{i\in\mathcal{T}_{s,<t}}$ where $\mathcal{T}_{s,<t}$
represents the set of timesteps of available news.

Considering the large sequence length that LLMs can process nowadays [47, 26],
we concatenate the set of news in the look-back window into one sequence
denoted by
$\mathbf{X}_{s,<t}=\oplus\\{\mathbf{x}_{s,i}\\}_{i\in\mathcal{T}_{s,<t}}$,
where $\oplus$ denotes the concatenation operation. Next, we formulate the
return forecasting model as a composite structure of a text representation
module and a forecasting module as defined in Eq. 1:

$\displaystyle\hat{r}_{s,t+\ell}=f\circ g\left(\mathbf{X}_{s,<t}\right)$ (1)

We aim to explore realizing Eq. 1 by jointly fine-tuning a pre-trained LLM as
$g(\cdot)$ and training a dense layer as $f(\cdot)$. In particular, Eq. 1 is a
sequence-level task requiring the text representation module
$g\colon\mathbf{X}_{s,<t}\mapsto\mathbf{h}_{s,<t}$ to encode the sequence
$\mathbf{X}_{s,<t}$ into a numerical vector
$\mathbf{h}_{s,<t}\in\mathbb{R}^{D}$. Then, the forecasting module
$f\colon\mathbf{h}_{s,<t}\mapsto\hat{r}_{s,t}$ transforms $\mathbf{h}_{s,<t}$
to the return forecast. We train the model using a set of data instances
pooled from individual stocks and associated news, i.e.,
$\\{(r_{s,t+\ell},\mathbf{X}_{s,<t})\\}_{s\in\mathcal{U},t\in\mathcal{T}}$
where $\mathcal{T}$ represents the timestamps in the training period.

At test time, besides evaluating prediction errors such as the root mean
square error (RMSE), we implement the return prediction-based stock picking to
construct long-only and long-short portfolios which are subsequently
backtested. This process is illustrated in Fig. 2.

Long-Only Portfolios is intended to include stocks with the expectation of a
price rise above the universe average. In practice, it is built by ranking the
stocks based on the return forecasts and selecting the top-K stocks. $K$ is
usually chosen according to the decile or quantile of the universe, e.g.,
$10\%$ of the total number of stocks.

Long-Short Portfolios includes both the stocks with the expectation of a price
rise and drop. For the stocks with a price drop expectation, the portfolio can
profit by selling them at the present price and repurchasing them at a lower
price in the future. In this paper, the long-short portfolio is built by
including the top-K and bottom-K stocks based on the forecast ranks.

###  3.2  Methodology

Transformer-based LLMs can be categorized into three main types: encoder-only,
decoder-only, and the hybrid encoder-decoder. All these LLMs transform text
into high-dimensional vector representations, however, their different pre-
training objectives lead to text representations with varying implications.

In the following, we describe the text representation difference in encoder-
only and decoder-only LLMs. Then, we present two simple methods of integrating
the token-level representations from LLMs into the forecasting module. These
methods introduce no additional parameters to learn and provide a clear
comparison of the native representations of different LLMs for return
forecasting.

Encoder-only LLMs vs. Decoder-only LLMs. Given a sequence of text tokens
$\mathbf{X}=\\{x_{1},\cdots,x_{L}\\}$, LLMs output a sequence of vector
representations $\\{\mathbf{h}_{1},\cdots,\mathbf{h}_{L}\\}$ corresponding to
the input tokens. However, as presented below, the vector representations from
encoder-only and decoder-only LLMs encode the different parts of the input
sequence.

Pre-training an encoder LLM is mostly based on masked-language modeling [9,
23, 15]. Concretely, it prepares a training text sequence $\mathbf{X}$ by
randomly masking some tokens, leading to
$\mathbf{\hat{X}}=\\{x_{\text{mask}}\,\,\text{if}\,i\in\mathcal{M}\,\,\text{else}\,\,x_{i}\,\,\forall\,i=1,\cdots,L\\}$.
$\mathcal{M}\subset\\{1,\cdots,L\\}$ represents the indices of tokens to mask.
The mask token $x_{\text{mask}}$ is a special token without concrete meaning
and plays as the placeholder. Then, the pre-training objective is to predict
the masked tokens, i.e., maximizing the likelihood of masked tokens as:

$\displaystyle\log
p\big{(}\\{x_{m}\\}_{m\in\mathcal{M}}\,|\,\mathbf{\hat{X}}\big{)}=\sum_{m\in\mathcal{M}}\log
p(x_{m}\,|\,\mathbf{X}_{<m},x_{\text{mask}},\mathbf{X}_{>m})\approx\sum_{m\in\mathcal{M}}\log
p(x_{m}\,|\,\mathbf{h}_{m})$ (2)

In Eq. 2, $\mathbf{X}_{<m}=\\{x_{1},\cdots,x_{m-1}\\}$ and
$\mathbf{X}_{>m}=\\{x_{m},\cdots,x_{L}\\}$ represent the tokens before and
after $x_{m}$. Maximizing Eq. 2 encourages the representation $\mathbf{h}_{m}$
to incorporate both the left and right contexts, i.e., $\mathbf{X}_{>m}$ and
$\mathbf{X}_{<m}$, for predicting the masked token. Particularly, in the
attention mechanism of Transformers, $\mathbf{h}_{m}$ is derived based on the
similarities between the mask token $x_{\text{mask}}$ and the context tokens
$\mathbf{X}_{>m}$ and $\mathbf{X}_{<m}$.

On the other hand, a decoder-only LLM models an input sequence
autoregressively using the next-token prediction task [31, 37]. The pre-
training objective function is defined in Eq. 3:

$\displaystyle\log
p(x_{1},\cdots,x_{L}|\mathbf{\check{X}})=\sum_{i=1,\cdots,L}\log
p(x_{i}\,|\,\mathbf{X}_{<i})\approx\sum_{i}\log p(x_{i}\,|\,\mathbf{h}_{i-1})$
(3)

For modeling the first token, the practical way is to add a Beginning-of-
Sequence (BOS) token, i.e.,
$\mathbf{\check{X}}=x_{\text{bos}}\oplus\mathbf{X}$. Similar to the mask
token, the BOS token has no concrete meaning. The representation
$\mathbf{h}_{i-1}$ encodes the information from already seen tokens and is
derived based on the relation between $x_{i-1}$ and
$\mathbf{X}_{<i-1}=\\{x_{1},\cdots,x_{i-2}\\}$.

Bottleneck Representations vs. Aggregated Representations. As LLMs output the
token-level vector representations, to obtain a representation encoding the
sequence, the idea of bottleneck representation is to push LLMs to compress
the sequence information into a single vector representation during fine-
tuning [46, 38, 39].

In practice, this is achieved by appending an End-of-Sequence (EOS)
${x}_{\text{EOS}}$ to the input sequence, e.g.,
$\mathbf{X}_{s,<t}\oplus{x}_{\text{EOS}}$. As ${x}_{\text{EOS}}$ is constant
across sequences, its vector representation $\mathbf{h}_{\text{EOS}}$ depends
on the real tokens of the sequence. During fine-tuning,
$\mathbf{h}_{\text{EOS}}$ is fed into the forecasting module as shown in Eq.
4. The backpropagation process propels $\mathbf{h}_{\text{EOS}}$ to summarize
real tokens’s representations through the forecasting module.

$\displaystyle\hat{r}_{s,t+\ell}=f(\mathbf{h}_{\text{EOS}})$ (4)

The bottleneck representation has different implications for encoder-only and
decoder-only LLMs. In encoder-only LLMs, the vector used for predicting is
obtained based on the mask token and the real context tokens during the pre-
training, as explained in Eq. 2. As a result, appending an EOS token
(identical to the mask token used in pre-training) aligns the fine-tuning with
the pre-training. This consistency might facilitate the EOS token
representation to summarize sequence-level features effectively. In decoder-
only LLMs, the vector representation of each token is conditioned on the
already-seen tokens; thus, the last token of a sequence naturally summarizes
the whole sequence, making an additional EOS token redundant.

In experiments, we observed that appending the EOS token is more helpful for
encoder-only LLMs. For a comparison on the same ground, we append EOS tokens
for both encoder-only and decoder-only LLMs and leave the study on the
different impacts of appending tokens to future work.

Meanwhile, considering the recent works on the representation collapse issue
of the last token in certain conditions [4], we present a simple alternative
to bottleneck representation, i.e., allowing the forecasting module to
aggregate the representations of all tokens. This can be done using various
methods like averaging, or sophisticated ones like attention mechanisms [24].
In this paper, we choose the simple averaging method, since it introduces no
additional parameters to train and enables a clear comparison with the
bottleneck representation.

$\displaystyle\hat{r}_{s,t+\ell}=f\left(\frac{1}{L}\sum_{l}\mathbf{h}_{l}\right)$
(5)

For encoder-only LLMs, the pre-training and fine-tuning discrepancy arises
when using aggregated representations, because each token’s representation is
based on context and itself, instead of the mask token in pre-training. For
decoder-only LLMs, averaging all representations might lead to bias towards
the early tokens of the input sequence. This is because, in the autoregressive
setting, the early tokens are repeatedly incorporated into the representations
of all subsequent ones.

Implementations. The text representation module and the forecasting module are
respectively initialized by a pre-trained LLM and a dense layer. Then, the
training process jointly fine-tunes the LLM and learns the forecasting module
to minimize the mean squared error (MSE) between the forecasts and true
values. We applied Low-Rank Adaptation (LoRA) to fine-tune LLMs [16]. Other
techniques including gradient checkpointing, mixed precision training, and
DeepSpeed are used to reduce GPU memory [34].

We experiment with one encoder-only LLM, i.e., DeBERTa [14], and two different
decoder-only LLMs, i.e., Mistral-7B and Llama3-8B base models [37, 19].
DeBERTa is a recent encoder-only LLM that improves upon the BERT model with
disentangled content and position embeddings. Mistral-7B is a 7-billion-
parameter decoder-only LLM that uses grouped query and sliding window
attention to improve performance. Llama3-8B is an 8-billion-parameter decoder-
only LLM pre-trained on data mixed from different sources, e.g., multilingual,
codes, etc., to improve the generalization ability.

##  4 Experiments

Data. We use company-level financial newsflow data from 2003 to 2019 provided
by a financial data vendor. Each piece of news has an attribute including the
company identifier(s) the news is primarily about. Meanwhile, we have two
investment universe datasets of the North American (NA), European (EU), and
Emerging (EM) markets, which consist of dates, stock identifiers, and the true
monthly forward returns of corresponding stocks and dates. The training and
validation data is from 2003 to 2014 for each universe, while the rest is for
the out-of-sample testing data. Each instance is built by linking an entry in
the universe data to related news through the stock identifier and a look-back
time window (e.g., one week). Table 2 shows the data stats.

Setup. We train the model only once and then apply the model to obtain the
return predictions in the testing period. We conduct the model training using
a batch size of $32$, a learning rate of $1e$-$5$, and a warmup phase of $100$
steps followed by a linear decay. To fine-tune LLMs, we applied Low-Rank
Adaptation (LoRA) with rank $4$ to all linear layers. We employ a maximum
context length of $4$k for all LLMs used in experiments. All models are
trained for $10$ epochs on $2$ A$100$ GPUs.

The long-only portfolio is built by taking the stocks with the return
predictions falling in the top ($9$th) decile of prediction rankings. The
long-short portfolios take the stocks in the top ($9$th) and bottom ($0$th)
deciles. The stocks in all portfolios are equally weighted.

We perform backtesting to evaluate the portfolios in monthly rebalancing. It
stimulates the trading of monthly constructed portfolios and reports the
cumulative return chart and performance statistics like annualized returns and
Sharpe ratios in the testing period. When backtesting the long-only and long-
short portfolios, besides comparing the portfolios built on return predictions
by different LLMs, we also compare them with the sentiment-based portfolio
construction. Specifically, FinBERT is a fine-tuned BERT (Bidirectional
Encoder Representations from Transformers) for financial sentiment analysis
[3]. FinVader is a dictionary-based method with a financial sentiment lexicon
[18, 22]. The sentiment-based portfolios are built using the same method but
with sentiment values as the ranking criteria.

Metrics. As mentioned in the problem statement of Sec. 3.1, the downstream
stock picking for building portfolios is based on the deciles of forecasts;
thus we report three decile-wise metrics to align with downstream scenarios,
i.e., decile RMSE, decile precision, and decile return. The decile return is
the actual return of stocks allocated to the decile based on predictions and
is directly related to the portfolio performance. Analyzing the decile return
along with the decile RMSE and precision provides insights into the relation
between portfolio performance and prediction accuracy.

Specifically, at each date in the testing data, we group the predictions with
the true returns into deciles based on the ranking of predictions (i.e., the
highest predictions are in the top $9$th decile and the lowest ones are in the
bottom 0th decile). Then, with the true and predicted returns in each decile
across dates, we calculate the decile RMSE, decile precision, and decile
return. The decile precision is the percentage of the true returns whose
decile based on the ranking of true values is equal to the current decile. It
is related to the portfolio performance, because, for instance, a high
precision of the top decile implies that a high proportion of stocks in this
decile has a high true forward return, thereby benefiting the portfolio
including stocks from the top decile.

For portfolio backtesting, we report the cumulative return charts and
performance statistics like annualized returns and Sharpe ratios in the
testing period.

Figure 3:  Decile Performance of Bottleneck and Aggregated Representations in
the North American Universe (best viewed in color). Top Row: Decile RMSE.
Middle Row: Decile Precision. Bottom Row: Decile Return. The up (or down)
arrow indicates the higher (or lower) values are desirable.  Table 1:
Statistics of Portfolios in the North American Universe. The Universe Equally-
Weighted represents the universe performance reported under the Long-only
Portfolio column.

| Long-only Portfolio | Long-short Portfolio
---|---|---
Ann. Return % ($\uparrow$) | Sharpe Ratio ($\uparrow$) | Ann. Return % ($\uparrow$) | Sharpe Ratio ($\uparrow$)
Universe Equally-Weighted | 9.76 | 0.68 | $-$ | $-$
Sentiment_FinVader | 12.26 | 0.72 | 2.92 | 0.39
Sentiment_FinBert | 20.64 | 1.22 | 8.81 | 0.92
DeBERTa_Bottleneck | 17.47 | 0.96 | 10.83 | 0.94
DeBERTa_Aggregated | 25.15 | 1.20 | 12.87 | 1.07
Mistral_Bottleneck | 21.27 | 1.15 | 15.08 | 1.49
Mistral_Aggregated | 25.38 | 1.12 | 18.30 | 1.26
Llama_Bottleneck | 27.00 | 1.32 | 20.46 | 1.49
Llama_Aggregated | 18.86 | 1.00 | 14.29 | 1.30

Results. In the following, we present and discuss mainly the results of the NA
universe. The results of the EU and EM universe are in the Appendix section.

Bottleneck Representations vs. Aggregated Representations: In Fig. 3, we
compare the bottleneck and aggregated representations for the three LLMs in
the North American universes through the decile RMSE, precision, and returns.
Each column of Fig. 3 corresponds to a LLM. Meanwhile, Fig. 4 shows the
cumulative return charts of portfolios and Table 1 reports the detailed
performance stats of portfolios.

In the bottom row of Fig. 3, the returns from the 0th decile to the 9th decile
generally present an upward trend, implying that overall the return
predictions are aligned with actual future performance. Moreover, we are
particularly interested in the top $9$th and bottom $0$th deciles as they are
the main constituents of portfolios. For the top $9$th decile, the aggregated
representation model generates a higher return and benefits the long
portfolio, except for Llama. For the EU and EM universe, as presented in the
Appendix section, the aggregated representation model consistently outperforms
the bottleneck one.

Interestingly, the higher returns do not necessarily imply low RMSE in the
$9$th decile. For instance, in Fig. 3, the aggregated representation model has
a higher decile return, but a higher RMSE, in the $9$th decile corresponding
to the long-only portfolio for DeBERTa and Mistral. An explanation is that the
$9$th decile is regarding predicting high-value returns and less accurate
predictions of these returns might have high RMSE. But, if the return
prediction still falls into the $9$th decile as the true return, the
corresponding decile return is retained. In this case, the decile precision is
more indicative of the decile return, for instance, in Fig. 3 the
outperforming representations mostly have a higher precision in the $9$th
decile.

As for the bottom $0$th decile, a lower return is preferred as the short side
of a long-short portfolio benefits from stocks with underperforming forward
returns. In Fig. 3, the aggregated representation model falls short of
lowering the 0th decile’s return for DeBERta and Mistral, however, Table 1
shows that the return and Sharpe ratios of long-short portfolios are mostly
improved with aggregated representations compared to the bottleneck
representations.

Meanwhile, in the 0th decile, there are complexities in how prediction errors
translate to actual returns. For instance, for DeBERTa, the aggregated
representation has higher RMSE and precision in the bottom $0$th decile,
implying that some stocks with higher true returns are misallocated to the 0th
decile by the prediction. As a result, the 0th decile return of the aggregated
representation is higher. However, when the aggregated representation of Llama
has the same pattern in the bottom decile, the return is as low as expected.
This might be because the high precision offsets the impact of misallocated
high returns.

Fig. 4 visualizes the cumulative return of the portfolios using the bottleneck
and aggregated representation models. The performance of long-only and long-
short portfolios correspond to the top and bottom deciles in Fig. 3. The
return curves of the aggregated representation model are notably higher except
for Llama. As shown in the Appendix, the aggregated representation constantly
outperforms the bottleneck representation for the EU and EM universes.

Figure 4:  Cumulative Return Charts of the Portfolios based on Bottleneck and
Aggregated Representation Models in the North American Universe (best viewed
in color). Top Row: Long-only Portfolios. Bottom Row: Long-short Portfolios.

Encoder-only LLMs vs. Decoder-only LLMs: Fig. 5 shows the comparison of
encoder-only and decoder-only LLMs with the suitable representations for the
NA universe, i.e., the aggregated representation for DeBERTa and Mistral, and
the bottleneck representation for Llama. For the EU and EM universes in the
Appendix, the aggregated representation is favored for all three LLMs.

The decile return in Fig. 5 exhibits that decoder-only Mistral and LLama
generate high returns in the top 9th decile and lower returns in the bottom
0th decile, thereby leading to the outperforming long-only and long-short
portfolios as shown in the cumulative return charts. In particular, the
performances of long-only portfolios are comparable among encoder and decoder
LLMs, however, in long-short portfolios, the short side drags down the
performance of the long side, especially for the encoder-only DeBERTa. This
highlights the importance of effective stock selection on both sides of the
portfolio. Meanwhile, all the prediction-based portfolios yield higher returns
than the universe average.

Figure 5:  Comparison of Encoder-only and Decoder-only LLMs with the Suited
Representations in the North American Universe (best viewed in color).

Prediction-based vs. Sentiment-based Portfolios: In this part, we compare the
prediction-based portfolios with conventional sentiment-based portfolios. Fig.
6 shows the decile returns and the return charts of portfolios, and the
performance statistics are in Table 1. The prediction-based portfolios are
from the forecasting model with the suited representations, as in the above
comparison of encoder-only and decoder-only LLMs.

In Table 1, the prediction-based long-only and long-short portfolios
outperform the sentiment-based portfolios both in returns and Sharp ratios. In
Fig. 6, the return charts of prediction-based portfolios are above the
sentiment-based portfolios. In particular, for the long-short portfolios, as
shown in the return chart, the short side of the sentiment-based method
negatively offsets the long side, leading to underperformance compared with
the universe. In contrast, the prediction-based long-short portfolios have
smoother return curves than the long-only portfolios, because the short side
mitigates the overall portfolio’s volatility. The outperformance of
prediction-based portfolios suggests that the return prediction models capture
more relevant information from text representations for future stock
performance, leading to effective stock picking.

Figure 6:  Comparison with Sentiment-based Portfolios in the North American
Universe (best viewed in color).

##  5 Conclusion

This paper focuses on return forecasting with financial newsflow for
quantitative portfolio construction. Unlike the conventional feature
extraction-and-validation workflow, this paper explores fine-tuning LLMs to
directly model the relationship between text representations and stock forward
return. Considering that different LLMs generate token-level representations
in distinct ways, we compare the design choices on two aspects: the encoder-
only versus decoder-only LLMs, and the bottleneck versus aggregated
representations.

Our experiments are conducted on real financial news, various investment
universes, and different portfolios. The results reveal the key findings: (1)
aggregated representations from LLMs’ token-level embeddings generally produce
the return predictions that enhance the performance of long-only and long-
short portfolios; (2) in the relatively large investment universe, the decoder
LLMs-based prediction model leads to stronger portfolios, whereas in the small
universes, there are no consistent winners. Among the three LLMs studied
(DeBERTa, Mistral, Llama), Mistral performs more robustly across different
universes; (3) return predictions derived from LLMs’ text representations are
a strong signal for portfolio construction, outperforming conventional
sentiment scores.

Several open questions remain for future research. For instance, it is unclear
whether the underperformance of encoder-only DeBERTa in the large investment
universe is due to the model size or other factors, and why DeBERTa has
varying performance in different small universes. Evaluating recently proposed
large encoder-only LLMs [39, 5] would be an interesting follow-up.
Additionally, within the decoder-only LLM family, compared with Mistral’s
robust performance across investment universes, the reasons behind Llama’s
performance variation need further exploration.

## References

  * [1] David E Allen, Michael McAleer, and Abhay K Singh.  Daily market news sentiment and stock prices.  Applied Economics, 51(30):3212–3235, 2019.
  * [2] Andrew Ang.  Asset management: A systematic approach to factor investing.  Oxford University Press, 2014.
  * [3] Dogu Araci.  Finbert: Financial sentiment analysis with pre-trained language models.  arXiv preprint arXiv:1908.10063, 2019.
  * [4] Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João GM Araújo, Alex Vitvitskyi, Razvan Pascanu, and Petar Veličković.  Transformers need glasses! information over-squashing in language tasks.  arXiv preprint arXiv:2406.04267, 2024.
  * [5] Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy.  Llm2vec: Large language models are secretly powerful text encoders.  arXiv preprint arXiv:2404.05961, 2024.
  * [6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [7] Deli Chen, Keiko Harimoto, Ruihan Bao, Qi Su, Xu Sun, et al.  Group, extract and aggregate: Summarizing a large amount of finance news for forex movement prediction.  arXiv preprint arXiv:1910.05032, 2019.
  * [8] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  Journal of Machine Learning Research, 25(70):1–53, 2024.
  * [9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019.
  * [10] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al.  Parameter-efficient fine-tuning of large-scale pre-trained language models.  Nature Machine Intelligence, 5(3):220–235, 2023.
  * [11] Eugene F Fama and Kenneth R French.  Multifactor explanations of asset pricing anomalies.  The journal of finance, 51(1):55–84, 1996.
  * [12] Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov.  Supervised contrastive learning for pre-trained language model fine-tuning.  arXiv preprint arXiv:2011.01403, 2020.
  * [13] Tian Guo, Nicolas Jamet, Valentin Betrix, Louis-Alexandre Piquet, and Emmanuel Hauptmann.  Esg2risk: A deep learning framework from esg news to stock volatility prediction.  arXiv preprint arXiv:2005.02527, 2020.
  * [14] Pengcheng He, Jianfeng Gao, and Weizhu Chen.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  arXiv preprint arXiv:2111.09543, 2021.
  * [15] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.  Deberta: Decoding-enhanced bert with disentangled attention.  arXiv preprint arXiv:2006.03654, 2020.
  * [16] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [17] Ziniu Hu, Weiqing Liu, Jiang Bian, Xuanzhe Liu, and Tie-Yan Liu.  Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction.  In Proceedings of the eleventh ACM international conference on web search and data mining, pages 261–269, 2018.
  * [18] Clayton Hutto and Eric Gilbert.  Vader: A parsimonious rule-based model for sentiment analysis of social media text.  In Proceedings of the international AAAI conference on web and social media, volume 8, pages 216–225, 2014.
  * [19] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  arXiv preprint arXiv:2310.06825, 2023.
  * [20] Alex Kim, Maximilian Muhn, and Valeri V Nikolaev.  Financial statement analysis with large language models.  Chicago Booth Research Paper Forthcoming, Fama-Miller Working Paper, 2024.
  * [21] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  Advances in neural information processing systems, 35:22199–22213, 2022.
  * [22] Petr Korab.  Finvader: Financial sentiment analysis, 2023.
  * [23] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations.  In International Conference on Learning Representations, 2019.
  * [24] Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping.  Nv-embed: Improved techniques for training llms as generalist embedding models.  arXiv preprint arXiv:2405.17428, 2024.
  * [25] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  arXiv preprint arXiv:1910.13461, 2019.
  * [26] Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen.  Large language models in finance: A survey.  In Proceedings of the fourth ACM international conference on AI in finance, pages 374–382, 2023.
  * [27] Qikai Liu, Xiang Cheng, Sen Su, and Shuguang Zhu.  Hierarchical complementary attention network for predicting stock price movements with news.  In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 1603–1606, 2018.
  * [28] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen.  Dora: Weight-decomposed low-rank adaptation.  arXiv preprint arXiv:2402.09353, 2024.
  * [29] Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and Jun Zhao.  Finbert: A pre-trained financial language representation model for financial text mining.  In Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence, pages 4513–4519, 2021.
  * [30] Yu Qin and Yi Yang.  What you say and how you say it matters: Predicting financial risk using verbal and vocal cues.  In 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019), page 390, 2019.
  * [31] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.  2018\.
  * [32] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [33] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of machine learning research, 21(140):1–67, 2020.
  * [34] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506, 2020.
  * [35] Ramit Sawhney, Shivam Agarwal, Arnav Wadhwa, and Rajiv Shah.  Deep attentive learning for stock movement prediction from social media text and company correlations.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8415–8426, 2020.
  * [36] Adam Hale Shapiro, Moritz Sudhof, and Daniel J Wilson.  Measuring news sentiment.  Journal of econometrics, 228(2):221–243, 2022.
  * [37] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [38] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei.  Simlm: Pre-training with representation bottleneck for dense passage retrieval.  In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.
  * [39] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei.  Improving text embeddings with large language models.  arXiv preprint arXiv:2401.00368, 2023.
  * [40] Meiyun Wang, Kiyoshi Izumi, and Hiroki Sakaji.  Llmfactor: Extracting profitable factors through prompts for explainable stock movement prediction.  arXiv preprint arXiv:2406.10811, 2024.
  * [41] Yaowei Wang, Qing Li, Zhexue Huang, and Junjie Li.  Ean: Event attention network for stock price trend prediction based on sentimental embedding.  In Proceedings of the 10th ACM Conference on Web Science, pages 311–320, 2019.
  * [42] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  arXiv preprint arXiv:2109.01652, 2021.
  * [43] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  Advances in neural information processing systems, 35:24824–24837, 2022.
  * [44] Bin Weng, Lin Lu, Xing Wang, Fadel M Megahed, and Waldyn Martinez.  Predicting short-term stock prices using ensemble methods and online data sources.  Expert Systems with Applications, 112:258–273, 2018.
  * [45] Yumo Xu and Shay B Cohen.  Stock movement prediction from tweets and historical prices.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1970–1979, 2018.
  * [46] Linyi Yang, Ruihai Dong, Tin Lok James Ng, and Yang Xu.  Leveraging bert to improve the fears index for stock forecasting.  In Proceedings of the First Workshop on Financial Technology and Natural Language Processing, pages 54–60, 2019.
  * [47] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al.  A survey of large language models.  arXiv preprint arXiv:2303.18223, 2023.

##  Appendix A Appendix

Table 2:  Statistics of Datasets.

Universe | # of Stocks | Average # of News per Instance | # of Training Instances | # of Validating Instances | # of Testing Instances
---|---|---|---|---|---
North America | 630 | 2.5 | 366011 | 10167 | 241367
Europe | 350 | 1.9 | 100403 | 10041 | 121705
Emerging Markets | 370 | 2.6 | 71610 | 10231 | 183608

###  A.1 Results of the European Universe

Figure 7:  Decile Performance of Bottleneck and Aggregated Representations in
the European Universe (best viewed in color). Top Row: Decile RMSE. Middle
Row: Decile Precision. Bottom Row: Decile Return. The up (or down) arrow
indicates the higher (or lower) values are desirable.  Table 3:  Statistics of
Portfolios in the European Universe. The Universe Equally-Weighted represents
the universe performance reported under the Long-only Portfolio column.

| Long-only Portfolio | Long-short Portfolio
---|---|---
Ann. Return % ($\uparrow$) | Sharpe Ratio ($\uparrow$) | Ann. Return % ($\uparrow$) | Sharpe Ratio ($\uparrow$)
Universe Equally-Weighted | 9.75 | 0.74 | $-$ | $-$
Sentiment_FinVader | 10.25 | 0.70 | 3.40 | 0.45
Sentiment_FinBert | 8.17 | 0.57 | -0.36 | 0.00
DeBERTa_Bottleneck | 11.04 | 0.81 | 2.11 | 0.31
DeBERTa_Aggregated | 11.11 | 0.81 | 3.84 | 0.52
Mistral_Bottleneck | 6.40 | 0.48 | 1.94 | 0.26
Mistral_Aggregated | 15.12 | 1.02 | 9.07 | 1.04
Llama_Bottleneck | 8.20 | 0.62 | 1.25 | 0.17
Llama_Aggregated | 12.76 | 0.90 | 11.47 | 1.27

Figure 8:  Cumulative Return Charts of the Portfolios based on Bottleneck and
Aggregated Representation Models in the European Universe (best viewed in
color). Top Row: Long-only Portfolios. Bottom Row: Long-short Portfolios.
Figure 9:  Comparison of Encoder-only and Decoder-only LLMs with the Suited
Representations in the European Universe (best viewed in color).  Figure 10:
Comparison with Sentiment-based Portfolios in the European Universe (best
viewed in color).

###  A.2 Results of the Emerging Markets Universe

Figure 11:  Decile Performance of Bottleneck and Aggregated Representations in
the Emerging Markets Universe (best viewed in color). Top Row: Decile RMSE.
Middle Row: Decile Precision. Bottom Row: Decile Return. The up (or down)
arrow indicates the higher (or lower) values are desirable.  Table 4:
Statistics of Portfolios in the Emerging Markets Universe. The Universe
Equally-Weighted represents the universe performance reported under the Long-
only Portfolio column.

| Long-only Portfolio | Long-short Portfolio
---|---|---
Ann. Return % ($\uparrow$) | Sharpe Ratio ($\uparrow$) | Ann. Return % ($\uparrow$) | Sharpe Ratio ($\uparrow$)
Universe Equally-Weighted | 3.91 | 0.32 | $-$ | $-$
Sentiment_FinVader | 6.18 | 0.43 | -0.08 | 0.04
Sentiment_FinBert | 9.76 | 0.70 | 1.69 | 0.21
DeBERTa_Bottleneck | 7.32 | 0.50 | -5.00 | -0.36
DeBERTa_Aggregated | 9.88 | 0.64 | 10.96 | 0.97
Mistral_Bottleneck | 10.12 | 0.63 | 4.94 | 0.47
Mistral_Aggregated | 10.11 | 0.64 | 9.16 | 0.68
Llama_Bottleneck | 4.94 | 0.36 | -3.99 | -0.28
Llama_Aggregated | 8.82 | 0.58 | 1.83 | 0.19

Figure 12:  Cumulative Return Charts of the Portfolios based on Bottleneck and
Aggregated Representation Models in the Emerging Markets Universe (best viewed
in color). Top Row: Long-only Portfolios. Bottom Row: Long-short Portfolios.

Figure 13:  Comparison of Encoder-only and Decoder-only LLMs with the Suited
Representations in the Emerging Markets Universe (best viewed in color).
Figure 14:  Comparison with Sentiment-based Portfolios in the Emerging Markets
Universe (best viewed in color).


# Human Motion Diffusion Model

Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir,
Daniel Cohen-Or and Amit H. Bermano
Tel Aviv University, Israel
<EMAIL_ADDRESS>

###### Abstract

Natural and expressive human motion generation is the holy grail of computer
animation. It is a challenging task, due to the diversity of possible motion,
human perceptual sensitivity to it, and the difficulty of accurately
describing it. Therefore, current generative solutions are either low-quality
or limited in expressiveness. Diffusion models, which have already shown
remarkable generative capabilities in other domains, are promising candidates
for human motion due to their many-to-many nature, but they tend to be
resource hungry and hard to control. In this paper, we introduce Motion
Diffusion Model (MDM), a carefully adapted classifier-free diffusion-based
generative model for the human motion domain. MDM is transformer-based,
combining insights from motion generation literature. A notable design-choice
is the prediction of the sample, rather than the noise, in each diffusion
step. This facilitates the use of established geometric losses on the
locations and velocities of the motion, such as the foot contact loss. As we
demonstrate, MDM is a generic approach, enabling different modes of
conditioning, and different generation tasks. We show that our model is
trained with lightweight resources and yet achieves state-of-the-art results
on leading benchmarks for text-to-motion and action-to-motion111Our code can
be found at https://github.com/GuyTevet/motion-diffusion-model..
https://guytevet.github.io/mdm-page/.

##  1 Introduction

Human motion generation is a fundamental task in computer animation, with
applications spanning from gaming to robotics. It is a challenging field, due
to several reasons, including the vast span of possible motions, and the
difficulty and cost of acquiring high quality data. For the recently emerging
text-to-motion setting, where motion is generated from natural language,
another inherent problem is data labeling. For example, the label ”kick” could
refer to a soccer kick, as well as a Karate one. At the same time, given a
specific kick there are many ways to describe it, from how it is performed to
the emotions it conveys, constituting a many-to-many problem. Current
approaches have shown success in the field, demonstrating plausible mapping
from text to motion (Petrovich et al., 2022; Tevet et al., 2022; Ahuja &
Morency, 2019). All these approaches, however, still limit the learned
distribution since they mainly employ auto-encoders or VAEs (Kingma & Welling,
2013) (implying a one-to-one mapping or a normal latent distribution
respectively). In this aspect, diffusion models are a better candidate for
human motion generation, as they are free from assumptions on the target
distribution, and are known for expressing well the many-to-many distribution
matching problem we have described.

Diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2020; Ho et al.,
2020) are a generative approach that is gaining significant attention in the
computer vision and graphics community. When trained for conditioned
generation, recent diffusion models (Ramesh et al., 2022; Saharia et al.,
2022b) have shown breakthroughs in terms of image quality and semantics. The
competence of these models have also been shown for other domains, including
videos (Ho et al., 2022), and 3D point clouds (Luo & Hu, 2021). The problem
with such models, however, is that they are notoriously resource demanding and
challenging to control.

In this paper, we introduce Motion Diffusion Model (MDM) — a carefully adapted
diffusion based generative model for the human motion domain. Being diffusion-
based, MDM gains from the native aforementioned many-to-many expression of the
domain, as evidenced by the resulting motion quality and diversity (Figure 1).
In addition, MDM combines insights already well established in the motion
generation domain, helping it be significantly more lightweight and
controllable.

\begin{overpic}[width=433.62pt,tics=10,trim=0.0pt 0.0pt 0.0pt
0.0pt,clip]{figures/teaser.pdf} \end{overpic} Figure 1:  Our Motion Diffusion
Model (MDM) reflects the many-to-many nature of text-to-motion mapping by
generating diverse motions given a text prompt. Our custom architecture and
geometric losses help yielding high-quality motion. Darker color indicates
later frames in the sequence.

First, instead of the ubiquitous U-net (Ronneberger et al., 2015) backbone,
MDM is transformer-based. As we demonstrate, our architecture (Figure 2) is
lightweight and better fits the temporal and non-spatial nature of motion data
(represented as a collection of joints). A large volume of motion generation
research is devoted to learning using geometric losses (Kocabas et al., 2020;
Harvey et al., 2020; Aberman et al., 2020). Some, for example, regulate the
velocity of the motion (Petrovich et al., 2021) to prevent jitter, or
specifically consider foot sliding using dedicated terms (Shi et al., 2020).
Consistently with these works, we show that applying geometric losses in the
diffusion setting improves generation.

The MDM framework has a generic design enabling different forms of
conditioning. We showcase three tasks: text-to-motion, action-to-motion, and
unconditioned generation. We train the model in a classifier-free manner (Ho &
Salimans, 2022), which enables trading-off diversity to fidelity, and sampling
both conditionally and unconditionally from the same model. In the text-to-
motion task, our model generates coherent motions (Figure 1) that achieve
state-of-the-art results on the HumanML3D (Guo et al., 2022a) and KIT
(Plappert et al., 2016) benchmarks. Moreover, our user study shows that human
evaluators prefer our generated motions over real motions $42\%$ of the time
(Figure 4(a)). In action-to-motion, MDM outperforms the state-of-the-art (Guo
et al., 2020; Petrovich et al., 2021), even though they were specifically
designed for this task, on the common HumanAct12 (Guo et al., 2020) and UESTC
(Ji et al., 2018) benchmarks.

Lastly, we also demonstrate completion and editing. By adapting diffusion
image-inpainting (Song et al., 2020b; Saharia et al., 2022a), we set a motion
prefix and suffix, and use our model to fill in the gap. Doing so under a
textual condition guides MDM to fill the gap with a specific motion that still
maintains the semantics of the original input. By performing inpainting in the
joints space rather than temporally, we also demonstrate the semantic editing
of specific body parts, without changing the others (Figure 3).

Overall, we introduce Motion Diffusion Model, a motion framework that achieves
state-of-the-art quality in several motion generation tasks, while requiring
only about three days of training on a single mid-range GPU. It supports
geometric losses, which are non trivial to the diffusion setting, but are
crucial to the motion domain, and offers the combination of state-of-the-art
generative power with well thought-out domain knowledge.

##  2 Related Work

###  2.1 Human Motion Generation

Neural motion generation, learned from motion capture data, can be conditioned
by any signal that describes the motion. Many works use parts of the motion
itself for guidance. Some predict motion from its prefix poses (Fragkiadaki et
al., 2015; Martinez et al., 2017; Hernandez et al., 2019; Guo et al., 2022b).
Others (Harvey & Pal, 2018; Kaufmann et al., 2020; Harvey et al., 2020; Duan
et al., 2021) solve in-betweening and super-resolution tasks using bi-
directional GRU (Cho et al., 2014) and Transformer (Vaswani et al., 2017)
architectures. Holden et al. (2016) use auto-encoder to learn motion latent
representation, then utilize it to edit and control motion with spatial
constraints such as root trajectory and bone lengths. Motion can be controlled
with a high-level guidance given from action class (Guo et al., 2020;
Petrovich et al., 2021; Cervantes et al., 2022), audio (Li et al., 2021;
Aristidou et al., 2022) and natural language (Ahuja & Morency, 2019; Petrovich
et al., 2022). In most cases authors suggests a dedicated approach to map each
conditioning domain into motion.

In recent years, the leading approach for the Text-to-Motion task is to learn
a shared latent space for language and motion. JL2P (Ahuja & Morency, 2019)
learns the KIT motion-language dataset (Plappert et al., 2016) with an auto-
encoder, limiting one-to-one mapping from text to motion. TEMOS (Petrovich et
al., 2022) and T2M (Guo et al., 2022a) suggest using a VAE (Kingma & Welling,
2013) to map a text prompt into a normal distribution in latent space.
Recently, MotionCLIP (Tevet et al., 2022) leverages the shared text-image
latent space learned by CLIP (Radford et al., 2021) to expand text-to-motion
out of the data limitations and enabled latent space editing.

The human motion manifold can also be learned without labels, as shown by
Holden et al. (2016), V-Poser (Pavlakos et al., 2019), and more recently the
dedicated MoDi architecture (Raab et al., 2022). We show that our model is
capable for such an unsupervised setting as well.

###  2.2 Diffusion Generative Models

Diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2020) are a class
of neural generative models, based on the stochastic diffusion process as it
is modeled in Thermodynamics. In this setting, a sample from the data
distribution is gradually noised by the diffusion process. Then, a neural
model learns the reverse process of gradually denoising the sample. Sampling
the learned data distribution is done by denoising a pure initial noise. Ho et
al. (2020) and Song et al. (2020a) further developed the practices for image
generation applications. For conditioned generation, Dhariwal & Nichol (2021),
introduced classifier-guided diffusion, which was later on adapted by GLIDE
(Nichol et al., 2021) to enable conditioning over CLIP textual
representations. The Classifier-Free Guidance approach Ho & Salimans (2022)
enables conditioning while trading-off fidelity and diversity, and achieves
better results (Nichol et al., 2021). In this paper, we implement text-to-
motion by conditioning on CLIP in a classifier-free manner, similarly to text-
to-image (Ramesh et al., 2022; Saharia et al., 2022b). Local editing of images
is typically defined as an inpainting problem, where a part of the image is
constant, and the inpainted part is denoised by the model, possibly under some
condition (Song et al., 2020b; Saharia et al., 2022a). We adapt this technique
to edit motion’s specific body parts or temporal intervals (in-betweening)
according to an optional condition.

More recently, concurrent to this work, Zhang et al. (2022) and Kim et al.
(2022) have suggested diffusion models for motion generation. Our work
requires significantly fewer GPU resources and makes design choices that
enable geometric losses, which improve results.

##  3 Motion Diffusion Model

\begin{overpic}[width=397.48499pt]{figures/mdm_comb.pdf} \end{overpic} Figure
2:  (Left) Motion Diffusion Model (MDM) overview. The model is fed a motion
sequence $x^{1:N}_{t}$ of length $N$ in a noising step $t$, as well as $t$
itself and a conditioning code $c$. $c$, a CLIP (Radford et al., 2021) based
textual embedding in this case, is first randomly masked for classifier-free
learning and then projected together with $t$ into the input token $z_{tk}$.
In each sampling step, the transformer-encoder predicts the final clean motion
$\hat{x}^{1:N}_{0}$. (Right) Sampling MDM. Given a condition $c$, we sample
random noise $x_{T}$ at the dimensions of the desired motion, then iterate
from $T$ to $1$. At each step $t$, MDM predicts the clean sample
$\hat{x}_{0}$, and diffuses it back to $x_{t-1}$.

An overview of our method is described in Figure 2. Our goal is to synthesize
a human motion $x^{1:N}$ of length $N$ given an arbitrary condition $c$. This
condition can be any real-world signal that will dictate the synthesis, such
as audio (Li et al., 2021; Aristidou et al., 2022), natural language (text-to-
motion) (Tevet et al., 2022; Guo et al., 2022a) or a discrete class (action-
to-motion) (Guo et al., 2020; Petrovich et al., 2021). In addition,
unconditioned motion generation is also possible, which we denote as the null
condition $c=\emptyset$. The generated motion $x^{1:N}=\\{x^{i}\\}_{i=1}^{N}$
is a sequences of human poses represented by either joint rotations or
positions $x^{i}\in\mathbb{R}^{J\times D}$, where $J$ is the number of joints
and $D$ is the dimension of the joint representation. MDM can accept motion
represented by either locations, rotations, or both (see Section 4).

Framework. Diffusion is modeled as a Markov noising process,
$\\{x^{1:N}_{t}\\}_{t=0}^{T}$, where $x^{1:N}_{0}$ is drawn from the data
distribution and

${q(x^{1:N}_{t}|x^{1:N}_{t-1})=\mathcal{N}(\sqrt{\alpha_{t}}x^{1:N}_{t-1},(1-\alpha_{t})I),}$
(1)

where $\alpha_{t}\in(0,1)$ are constant hyper-parameters. When $\alpha_{t}$ is
small enough, we can approximate $x^{1:N}_{T}\sim\mathcal{N}(0,I)$. From here
on we use $x_{t}$ to denote the full sequence at noising step $t$.

In our context, conditioned motion synthesis models the distribution
$p(x_{0}|c)$ as the reversed diffusion process of gradually cleaning $x_{T}$.
Instead of predicting $\epsilon_{t}$ as formulated by Ho et al. (2020), we
follow Ramesh et al. (2022) and predict the signal itself, i.e.,
$\hat{x}_{0}=G(x_{t},t,c)$ with the simple objective (Ho et al., 2020),

${\mathcal{L}_{\text{simple}}=E_{x_{0}\sim
q(x_{0}|c),t\sim[1,T]}[\|x_{0}-G(x_{t},t,c)\|_{2}^{2}]}$ (2)

Geometric losses. In the motion domain, generative networks are standardly
regularized using geometric losses Petrovich et al. (2021); Shi et al. (2020).
These losses enforce physical properties and prevent artifacts, encouraging
natural and coherent motion. In this work we experiment with three common
geometric losses that regulate (1) positions (in case we predict rotations),
(2) foot contact, and (3) velocities.

${\mathcal{L}_{\text{pos}}=\frac{1}{N}\sum_{i=1}^{N}\|FK(x^{i}_{0})-FK(\hat{x}^{i}_{0})\|_{2}^{2},}$
(3)

${\mathcal{L}_{\text{foot}}=\frac{1}{N-1}\sum_{i=1}^{N-1}\|(FK(\hat{x}^{i+1}_{0})-FK(\hat{x}^{i}_{0}))\cdot
f_{i}\|_{2}^{2},}$ (4)

${\mathcal{L}_{\text{vel}}=\frac{1}{N-1}\sum_{i=1}^{N-1}\|(x^{i+1}_{0}-x^{i}_{0})-(\hat{x}^{i+1}_{0}-\hat{x}^{i}_{0})\|_{2}^{2}}$
(5)

In case we predict joint rotations, $FK(\cdot)$ denotes the forward kinematic
function converting joint rotations into joint positions (otherwise, it
denotes the identity function). $f_{i}\in\\{0,1\\}^{J}$ is the binary foot
contact mask for each frame $i$. Relevant only to feet, it indicates whether
they touch the ground, and are set according to binary ground truth data (Shi
et al., 2020). In essence, it mitigates the foot-sliding effect by nullifying
velocities when touching the ground.

Overall, our training loss is

${\mathcal{L}=\mathcal{L}_{\text{simple}}+\lambda_{\text{pos}}\mathcal{L}_{\text{pos}}+\lambda_{\text{vel}}\mathcal{L}_{\text{vel}}+\lambda_{\text{foot}}\mathcal{L}_{\text{foot}}.}$
(6)

Model. Our model is illustrated in Figure 2. We implement $G$ with a
straightforward transformer (Vaswani et al., 2017) encoder-only architecture.
The transformer architecture is temporally aware, enabling learning arbitrary
length motions, and is well-proven for the motion domain (Petrovich et al.,
2021; Duan et al., 2021; Aksan et al., 2021). The noise time-step $t$ and the
condition code $c$ are each projected to the transformer dimension by separate
feed-forward networks, then summed to yield the token $z_{tk}$. Each frame of
the noised input $x_{t}$ is linearly projected into the transformer dimension
and summed with a standard positional embedding. $z_{tk}$ and the projected
frames are then fed to the encoder. Excluding the first output token
(corresponding to $z_{tk}$), the encoder result is projected back to the
original motion dimensions, and serves as the prediction $\hat{x}_{0}$. We
implement text-to-motion by encoding the text prompt to $c$ with CLIP (Radford
et al., 2021) text encoder, and action-to-motion with learned embeddings per
class.

Sampling from $p(x_{0}|c)$ is done in an iterative manner, according to Ho et
al. (2020). In every time step $t$ we predict the clean sample
$\hat{x}_{0}=G(x_{t},t,c)$ and noise it back to $x_{t-1}$. This is repeated
from $t=T$ until $x_{0}$ is achieved (Figure 2 right). We train our model $G$
using classifier-free guidance (Ho & Salimans, 2022). In practice, $G$ learns
both the conditioned and the unconditioned distributions by randomly setting
$c=\emptyset$ for $10\%$ of the samples, such that $G(x_{t},t,\emptyset)$
approximates $p(x_{0})$. Then, when sampling $G$ we can trade-off diversity
and fidelity by interpolating or even extrapolating the two variants using
$s$:

${G_{s}(x_{t},t,c)=G(x_{t},t,\emptyset)+s\cdot(G(x_{t},t,c)-G(x_{t},t,\emptyset))}$
(7)

Editing. We enable motion in-betweening in the temporal domain, and body part
editing in the spatial domain, by adapting diffusion inpainting to motion
data. Editing is done only during sampling, without any training involved.
Given a subset of the motion sequence inputs, when sampling the model (Figure
2 right), at each iteration we overwrite $\hat{x}_{0}$ with the input part of
the motion. This encourages the generation to remain coherent to original
input, while completing the missing parts. In the temporal setting, the prefix
and suffix frames of the motion sequence are the input, and we solve a motion
in-betweening problem (Harvey et al., 2020). Editing can be done either
conditionally or unconditionally (by setting $c=\emptyset$). In the spatial
setting, we show that body parts can be re-synthesized according to a
condition $c$ while keeping the rest intact, through the use of the same
completion technique.

##  4 Experiments

\begin{overpic}[width=397.48499pt]{figures/edit.pdf} \end{overpic} Figure 3:
Editing applications. Light blue frames represent motion input and bronze
frames are the generated motion. Motion in-betweening (left+center) can be
performed conditioned on text or without condition by the same model. Specific
body part editing using text is demonstrated on the right: the lower body
joints are fixed to the input motion while the upper body is altered to fit
the input text prompt.

We implement MDM for three motion generation tasks: Text-to-Motion(4.1),
Action-to-Motion(4.2) and unconditioned generation(5.2. Each sub-section
reviews the data and metrics of the used benchmarks, provides implementation
details, and presents qualitative and quantitative results. Then, we show
implementations of motion in-betweening (both conditioned and unconditioned)
and body-part editing by adapting diffusion inpainting to motion (5.1). Our
models have been trained with $T=1000$ noising steps and a cosine noise
schedule. All of them have been trained on a single NVIDIA GeForce RTX 2080 Ti
GPU for a period of about $3$ days.

###  4.1 Text-to-Motion

Text-to-motion is the task of generating motion given an input text prompt.
The output motion is expected to be both implementing the textual description,
and a valid sample from the data distribution (i.e. adhering to general human
abilities and the rules of physics). In addition, for each text prompt, we
also expect a distribution of motions matching it, rather than just a single
result. We evaluate our model using two leading benchmarks - KIT (Plappert et
al., 2016) and HumanML3D (Guo et al., 2022a), over the set of metrics
suggested by Guo et al. (2022a): R-precision and Multimodal-Dist measure the
relevancy of the generated motions to the input prompts, FID measures the
dissimilarity between the generated and ground truth distributions (in latent
space), Diversity measures the variability in the resulting motion
distribution, and MultiModality is the average variance given a single text
prompt. For the full implementation of the metrics, please refer to Guo et al.
(2022a). We use HumanML3D as a platform to compare different backbones of our
model, discovering that the diffusion framework is relatively agnostic to this
attribute. In addition, we conduct a user study comparing our model to current
art and ground truth motions.

Data. HumanML3D is a recent dataset, textually re-annotating motion capture
from the AMASS (Mahmood et al., 2019) and HumanAct12 (Guo et al., 2020)
collections. It contains $14,616$ motions annotated by $44,970$ textual
descriptions. In addition, it suggests a redundant data representation
including a concatenation of root velocity, joint positions, joint velocities,
joint rotations and the foot contact binary labels. We also use in this
section the same representation for the KIT dataset, brought by the same
publishers. Although limited in the number ($3,911$) and the diversity of
samples, most of the text-to-motion research is based on KIT, hence we view it
as important to evaluate using it as well.

Implementation. In addition to our Transformer encoder-only backbone (Section
3), we experiment MDM with three more backbones: (1) Transformer decoder
injects $z_{tk}$ through the cross-attention layer, instead of as an input
token. (2) Transformer decoder + input token, where $z_{tk}$ is injected both
ways, and (3) GRU (Cho et al., 2014) concatenate $z_{tk}$ to each input frame
(Table 1). Our models were trained with batch size $64$, $8$ layers (except
GRU that was optimal at $2$), and latent dimension $512$. To encode the text
we use a frozen CLIP-ViT-B/32 model. Each model was trained for $500K$ steps,
afterwhich a checkpoint was chosen that minimizes the FID metric to be
reported. Since foot contact and joint locations are explicitly represented in
HumanML3D, we don’t apply geometric losses in this section. We evaluate our
models with guidance-scale $s=2.5$ which provides a diversity-fidelity sweet
spot (Figure 4).

Quantitative evaluation. We evaluate and compare our models to current art
(JL2P Ahuja & Morency (2019), Text2Gesture (Bhattacharya et al., 2021), and
T2M (Guo et al., 2022a)) with the metrics suggested by Guo et al. (2022a). As
can be seen, MDM achieves state-of-the-art results in FID, Diversity, and
MultiModality, indicating high diversity per input text prompt, and high-
quality samples, as can also be seen qualitatively in Figure 1.

User study. We asked 31 users to choose between MDM and state-of-the-art works
in a side-by-side view, with both samples generated from the same text prompt
randomly sampled from the KIT test set. We repeated this process with 10
samples per model and 10 repetitions per sample. This user study enabled a
comparison with the recent TEMOS model (Petrovich et al., 2022), which was not
included in the HumanML3D benchmark. Fig. 4 shows that most of the time, MDM
was preferred over the compared models, and even preferred over ground truth
samples in $42.3\%$ of the cases.

Method | R Precision (top 3)$\uparrow$ | FID$\downarrow$ | Multimodal Dist$\downarrow$ | Diversity$\rightarrow$ | Multimodality$\uparrow$
---|---|---|---|---|---
Real | $0.797^{\pm.002}$ | $0.002^{\pm.000}$ | $2.974^{\pm.008}$ | $9.503^{\pm.065}$ | -
JL2P | $0.486^{\pm.002}$ | $11.02^{\pm.046}$ | $5.296^{\pm.008}$ | $7.676^{\pm.058}$ | -
Text2Gesture | $0.345^{\pm.002}$ | $7.664^{\pm.030}$ | $6.030^{\pm.008}$ | $6.409^{\pm.071}$ | -
T2M | $\mathbf{0.740^{\pm.003}}$ | $1.067^{\pm.002}$ | $\mathbf{3.340^{\pm.008}}$ | $9.188^{\pm.002}$ | $2.090^{\pm.083}$
MDM (ours) | ${0.611^{\pm.007}}$ | $\mathbf{0.544^{\pm.044}}$ | ${5.566^{\pm.027}}$ | $\mathbf{9.559^{\pm.086}}$ | ${2.799^{\pm.072}}$
MDM (decoder) | ${0.608^{\pm.005}}$ | ${0.767^{\pm.085}}$ | ${5.507^{\pm.020}}$ | ${9.176^{\pm.070}}$ | $\mathbf{2.927^{\pm.125}}$
\+ input token | ${0.621^{\pm.005}}$ | ${0.567^{\pm.051}}$ | ${5.424^{\pm.022}}$ | ${9.425^{\pm.060}}$ | $2.834^{\pm.095}$
MDM (GRU) | ${0.645^{\pm.005}}$ | ${4.569^{\pm.150}}$ | ${5.325^{\pm.026}}$ | ${7.688^{\pm.082}}$ | $1.2646^{\pm.024}$

Table 1: Quantitative results on the HumanML3D test set. All methods use the
real motion length from the ground truth. ‘$\rightarrow$’ means results are
better if the metric is closer to the real distribution. We run all the
evaluation 20 times (except MultiModality runs 5 times) and $\pm$ indicates
the 95% confidence interval. Bold indicates best result.

Method | R Precision (top 3)$\uparrow$ | FID$\downarrow$ | Multimodal Dist$\downarrow$ | Diversity$\rightarrow$ | Multimodality$\uparrow$
---|---|---|---|---|---
Real | $0.779^{\pm.006}$ | $0.031^{\pm.004}$ | $2.788^{\pm.012}$ | $11.08^{\pm.097}$ | -
JL2P | $0.483^{\pm.005}$ | $6.545^{\pm.072}$ | $5.147^{\pm.030}$ | $9.073^{\pm.100}$ | -
Text2Gesture | $0.338^{\pm.005}$ | $12.12^{\pm.183}$ | $6.964^{\pm.029}$ | $9.334^{\pm.079}$ | -
T2M | $\mathbf{0.693^{\pm.007}}$ | ${2.770^{\pm.109}}$ | $\mathbf{3.401^{\pm.008}}$ | $\mathbf{10.91^{\pm.119}}$ | $1.482^{\pm.065}$
MDM (ours) | ${0.396^{\pm.004}}$ | $\mathbf{0.497^{\pm.021}}$ | ${9.191^{\pm.022}}$ | ${10.847^{\pm.109}}$ | $\mathbf{1.907^{\pm.214}}$

Table 2: Quantitative results on the KIT test set.

(a) KIT User Study (b) Classifier-free scale sweep

Figure 4: (a) Text-to-motion user study for the KIT dataset. Each bar
represents the preference rate of MDM over the compared model. MDM was
preferred over the other models in most of the time, and $42.3\%$ of the cases
even over ground truth samples. The dashed line marks $50\%$. (b) Guidance-
scale sweep for HumanML3D dataset. FID (lower is better) and R-precision
(higher is better) metrics as a function of the scale $s$, draws an accuracy-
fidelity sweet spot around $s=2.5$.

Method | FID$\downarrow$ | Accuracy$\uparrow$ | Diversity$\rightarrow$ | Multimodality$\rightarrow$
---|---|---|---|---
Real (INR) | $0.020^{\pm.010}$ | $0.997^{\pm.001}$ | $6.850^{\pm.050}$ | $2.450^{\pm.040}$
Real (ours) | $0.050^{\pm.000}$ | $0.990^{\pm.000}$ | $6.880^{\pm.020}$ | $2.590^{\pm.010}$
Action2Motion (2020) | $0.338^{\pm.015}$ | $0.917^{\pm.003}$ | $6.879^{\pm.066}$ | $\underline{2.511^{\pm.023}}$
ACTOR (2021) | $0.120^{\pm.000}$ | $0.955^{\pm.008}$ | $\mathbf{6.840^{\pm.030}}$ | $2.530^{\pm.020}$
INR (2022) | $\underline{0.088^{\pm.004}}$ | $\underline{0.973^{\pm.001}}$ | $6.881^{\pm.048}$ | $2.569^{\pm.040}$
MDM (ours) | $0.100^{\pm.000}$ | $\mathbf{0.990^{\pm.000}}$ | $\underline{6.860^{\pm.050}}$ | $2.520^{\pm.010}$
w/o foot contact | $\mathbf{0.080^{\pm.000}}$ | $\mathbf{0.990^{\pm.000}}$ | $6.810^{\pm.010}$ | $\mathbf{2.580^{\pm.010}}$

Table 3: Evaluation of action-to-motion on the HumanAct12 dataset. Our model
leads the board in three out of four metrics. Ground-truth evaluation results
are slightly different for each of the works, due to implementation
differences, such as python package versions. It is important to assess the
diversity and multimodality of each model using its own ground-truth results,
as they are measured by their distance from GT. We show the GT metrics
measured by our model and by the leading compared work, INR (Cervantes et al.,
2022). Bold indicates best result, $\underline{\text{underline}}$ indicates
second best, $\pm$ indicates 95% confidence interval, $\rightarrow$ indicates
that closer to real is better.

Method | $\text{FID}_{\text{train}}\downarrow$ | $\text{FID}_{\text{test}}\downarrow$ | Accuracy$\uparrow$ | Diversity$\rightarrow$ | Multimodality$\rightarrow$
---|---|---|---|---|---
Real | $2.92^{\pm.26}$ | $2.79^{\pm.29}$ | $0.988^{\pm.001}$ | $33.34^{\pm.320}$ | $14.16^{\pm.06}$
ACTOR (2021) | $20.49^{\pm 2.31}$ | $23.43^{\pm 2.20}$ | $0.911^{\pm.003}$ | $31.96^{\pm.33}$ | $14.52^{\pm.09}$
INR (2022) (best variation) | $\mathbf{9.55^{\pm.06}}$ | $15.00^{\pm.09}$ | $0.941^{\pm.001}$ | $31.59^{\pm.19}$ | $14.68^{\pm.07}$
MDM (ours) | $9.98^{\pm 1.33}$ | $\mathbf{12.81^{\pm 1.46}}$ | $\underline{0.950^{\pm.000}}$ | $\underline{33.02^{\pm.28}}$ | $\mathbf{14.26^{\pm.12}}$
w/o foot contact | $\underline{9.69^{\pm.81}}$ | $\underline{13.08^{\pm 2.32}}$ | $\mathbf{0.960^{\pm.000}}$ | $\mathbf{33.10^{\pm.29}}$ | $\mathbf{14.06^{\pm.05}}$

Table 4: Evaluation of action-to-motion on the UESTC dataset. The performance
improvement with our model shows a clear gap from state-of-the-art. Bold
indicates best result, $\underline{\text{underline}}$ indicates second best,
$\pm$ indicates 95% confidence interval, $\rightarrow$ indicates that closer
to real is better.

###  4.2 Action-to-Motion

Action-to-motion is the task of generating motion given an input action class,
represented by a scalar. The output motion should faithfully animate the input
action, and at the same time be natural and reflect the distribution of the
dataset on which the model is trained. Two dataset are commonly used to
evaluate action-to-motion models: HumanAct12 (Guo et al., 2020) and UESTC (Ji
et al., 2018). We evaluate our model using the set of metrics suggested by Guo
et al. (2020), namely Fréchet Inception Distance (FID), action recognition
accuracy, diversity and multimodality. The combination of these metrics makes
a good measure of the realism and diversity of generated motions.

Data. HumanAct12 (Guo et al., 2020) offers approximately 1200 motion clips,
organized into 12 action categories, with 47 to 218 samples per label. UESTC
(Ji et al., 2018) consists of 40 action classes, 40 subjects and 25K samples,
and is split to train and test. We adhere to the cross-subject testing
protocol used by current works, with 225-345 samples per action class. For
both datasets we use the sequences provided by Petrovich et al. (2021).

Implementation. The implementation presented in Figure 2 holds for all the
variations of our work. In the case of action-to-motion, the only change would
be the substitution of the text embedding by an action embedding. Since action
is represented by a scalar, its embedding is fairly simple; each input action
class scalar is converted into a learned embedding of the transformer
dimension.

The experiments have been run with batch size $64$, a latent dimension of
$512$, and an encoder-transformer architecture. Training on HumanAct12 and
UESTC has been carried out for $750K$ and $2M$ steps respectively. In our
tables we display the evaluation of the checkpoint that minimizes the FID
metric.

Quantitative evaluation. Tables 3 and 4 reflect MDM’s performance on the
HumanAct12 and UESTC datasets respectively. We conduct 20 evaluations, with
1000 samples in each, and report their average and a 95% confidence interval.
We test two variations, with and without foot contact loss. Our model leads
the board for both datasets. The variation with no foot contact loss attains
slightly better results; nevertheless, as shown in our supplementary video,
the contribution of foot contact loss to the quality of results is important,
and without it we witness artifacts such as shakiness and unnatural gestures.

##  5 Additional Applications

###  5.1 Motion Editing

In this section we implement two motion editing applications - in-betweening
and body part editing, both using the same approach in the temporal and
spatial domains correspondingly. For in-betweening, we fix the first and last
$25\%$ of the motion, leaving the model to generate the remaining $50\%$ in
the middle. For body part editing, we fix the joints we don’t want to edit and
leave the model to generate the rest. In particular, we experiment with
editing the upper body joints only. In figure 3 we show that in both cases,
using the method described in Section 3 generates smooth motions that adhere
both to the fixed part of the motion and the condition (if one was given).

Method | FID$\downarrow$ | KID$\downarrow$ | Precision$\uparrow$ Recall$\uparrow$ | Multimodality$\uparrow$
---|---|---|---|---
ACTOR (2021) | 48.80 | 0.53 | 0.72, 0.74 | 14.10
MoDi (2022) | 13.03 | 0.12 |  0.71, 0.81 | 17.57
MDM (ours) | 31.92 | 0.36 | 0.66, 0.62 | 17.00
Table 5: Evaluation of unconstrained synthesis on the HumanAct12 dataset. We
test MDM in the challenging unconstrained setting, and compare with MoDi (Raab
et al., 2022), a work that was specially designed for such setting. We
demonstrate that in addition to being able to support any condition, we can
achieve plausible results in the unconstrained setting. Bold indicates best
result.

###  5.2 Unconstrained Synthesis

The challenging task of unconstrained synthesis has been studied by only a few
(Holden et al., 2016; Raab et al., 2022). In the presence of data labeling,
e.g., action classes or text description, the labels work as a supervising
factor, and facilitate a structured latent space for the training network. The
lack of labeling make training more difficult. The human motion field
possesses rich unlabeled datasets (Adobe Systems Inc., 2021), and the ability
to train on top of them is an advantage. Daring to test MDM in the challenging
unconstrained setting, we follow MoDi(Raab et al., 2022) for evaluation. We
use the metrics they suggest (FID, KID, precision/recall and multimodality),
and run on an unconstrained version of the HumanAct12 (Guo et al., 2020)
dataset.

Data. Although annotated, we use HumanAct12 (see Section 4.2) in an
unconstrained fashion, ignoring its labels. The choice of HumanAct12 rather
than a dataset with no labels (e.g., Mixamo (Adobe Systems Inc., 2021)), is
for compatibility with previous publications.

Implementation. Our model uses the same architecture for all forms of
conditioning, as well as for the unconstrained setting. The only change to the
structure shown in Figure 2, is the removal of the conditional input, such
that $z_{tk}$ is composed of the projection of $t$ only. To simulate an
unconstrained behavior, ACTOR Petrovich et al. (2021) has been trained by
(Raab et al., 2022) with a labeling of one class to all motions.

Quantitative evaluation. The results of our evaluation are shown in table 5.
We demonstrate superiority over works that were not designed for an
unconstrained setting, and get closer to MoDi (Raab et al., 2022). MoDi is
carefully molded for unconstrained settings, while our work can be applied to
any (or no) constrain, and also provides editing capabilities.

##  6 Discussion

We have presented MDM, a method that lends itself to various human motion
generation tasks. MDM is an untypical classifier-free diffusion model,
featuring a transformer-encoder backbone, and predicting the signal, rather
than the noise. This yields both a lightweight model, that is unburdening to
train, and an accurate one, gaining much from the applicable geometric losses.
Our experiments show superiority in conditioned generation, but also that this
approach is not very sensitive to the choice of architecture.

A notable limitation of the diffusion approach is the long inference time,
requiring about $1000$ forward passes for a single result. Since our motion
model is small anyway, using dimensions order of magnitude smaller than
images, our inference time shifts from less than a second to only about a
minute, which is an acceptable compromise. As diffusion models continue to
evolve, beside better compute, in the future we would be interested in seeing
how to incorporate better control into the generation process, and widen the
options for applications even further.

## Acknowledgements

We thank Rinon Gal for his useful suggestions and references. This research
was supported in part by the Israel Science Foundation (grants no. 2492/20 and
3441/21), Len Blavatnik and the Blavatnik family foundation, and The Tel Aviv
University Innovation Laboratories (TILabs).

## References

  * Aberman et al. (2020) Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Sorkine-Hornung, Daniel Cohen-Or, and Baoquan Chen.  Skeleton-aware networks for deep motion retargeting.  _ACM Transactions on Graphics (TOG)_ , 39(4):62–1, 2020.
  * Adobe Systems Inc. (2021) Adobe Systems Inc.  Mixamo, 2021.  URL https://www.mixamo.com.  Accessed: 2021-12-25.
  * Ahuja & Morency (2019) Chaitanya Ahuja and Louis-Philippe Morency.  Language2pose: Natural language grounded pose forecasting.  In _2019 International Conference on 3D Vision (3DV)_ , pp. 719–728. IEEE, 2019.
  * Aksan et al. (2021) Emre Aksan, Manuel Kaufmann, Peng Cao, and Otmar Hilliges.  A spatio-temporal transformer for 3d human motion prediction.  In _2021 International Conference on 3D Vision (3DV)_ , pp. 565–574. IEEE, 2021.
  * Aristidou et al. (2022) A Aristidou, A Yiannakidis, K Aberman, D Cohen-Or, A Shamir, and Y Chrysanthou.  Rhythm is a dancer: Music-driven motion synthesis with global structure.  _IEEE Transactions on Visualization and Computer Graphics_ , 2022\.
  * Bhattacharya et al. (2021) Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha.  Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents.  In _2021 IEEE Virtual Reality and 3D User Interfaces (VR)_ , pp. 1–10. IEEE, 2021.
  * Cervantes et al. (2022) Pablo Cervantes, Yusuke Sekikawa, Ikuro Sato, and Koichi Shinoda.  Implicit neural representations for variable length human motion generation.  _arXiv preprint arXiv:2203.13694_ , 2022.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_ , 2014.
  * Dhariwal & Nichol (2021) Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems_ , 34:8780–8794, 2021.
  * Duan et al. (2021) Yinglin Duan, Tianyang Shi, Zhengxia Zou, Yenan Lin, Zhehui Qian, Bohan Zhang, and Yi Yuan.  Single-shot motion completion with transformer.  _arXiv preprint arXiv:2103.00776_ , 2021.
  * Fragkiadaki et al. (2015) Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik.  Recurrent network models for human dynamics.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 4346–4354, 2015.
  * Guo et al. (2020) Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng.  Action2motion: Conditioned generation of 3d human motions.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pp. 2021–2029, 2020.
  * Guo et al. (2022a) Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng.  Generating diverse and natural 3d human motions from text.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5152–5161, 2022a.
  * Guo et al. (2022b) Wen Guo, Yuming Du, Xi Shen, Vincent Lepetit, Xavier Alameda-Pineda, and Francesc Moreno-Noguer.  Back to mlp: A simple baseline for human motion prediction.  _arXiv preprint arXiv:2207.01567_ , 2022b.
  * Harvey & Pal (2018) Félix G Harvey and Christopher Pal.  Recurrent transition networks for character locomotion.  In _SIGGRAPH Asia 2018 Technical Briefs_ , pp. 1–4. 2018.
  * Harvey et al. (2020) Félix G Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal.  Robust motion in-betweening.  _ACM Transactions on Graphics (TOG)_ , 39(4):60–1, 2020.
  * Hernandez et al. (2019) Alejandro Hernandez, Jurgen Gall, and Francesc Moreno-Noguer.  Human motion prediction via spatio-temporal inpainting.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 7134–7143, 2019.
  * Ho & Salimans (2022) Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Ho et al. (2022) Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  _arXiv preprint arXiv:2204.03458_ , 2022.
  * Holden et al. (2016) Daniel Holden, Jun Saito, and Taku Komura.  A deep learning framework for character motion synthesis and editing.  _ACM Transactions on Graphics (TOG)_ , 35(4):1–11, 2016.
  * Ji et al. (2018) Yanli Ji, Feixiang Xu, Yang Yang, Fumin Shen, Heng Tao Shen, and Wei-Shi Zheng.  A large-scale rgb-d database for arbitrary-view human action recognition.  In _Proceedings of the 26th ACM international Conference on Multimedia_ , pp. 1510–1518, 2018.
  * Kaufmann et al. (2020) Manuel Kaufmann, Emre Aksan, Jie Song, Fabrizio Pece, Remo Ziegler, and Otmar Hilliges.  Convolutional autoencoders for human motion infilling.  In _2020 International Conference on 3D Vision (3DV)_ , pp. 918–927. IEEE, 2020.
  * Kim et al. (2022) Jihoon Kim, Jiseob Kim, and Sungjoon Choi.  Flame: Free-form language-based motion synthesis & editing.  _arXiv preprint arXiv:2209.00349_ , 2022.
  * Kingma & Welling (2013) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kocabas et al. (2020) Muhammed Kocabas, Nikos Athanasiou, and Michael J Black.  Vibe: Video inference for human body pose and shape estimation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 5253–5263, 2020.
  * Li et al. (2021) Ruilong Li, Shan Yang, David A. Ross, and Angjoo Kanazawa.  Ai choreographer: Music conditioned 3d dance generation with aist++.  In _The IEEE International Conference on Computer Vision (ICCV)_ , 2021.
  * Luo & Hu (2021) Shitong Luo and Wei Hu.  Diffusion probabilistic models for 3d point cloud generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2837–2845, 2021.
  * Mahmood et al. (2019) Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black.  AMASS: Archive of motion capture as surface shapes.  In _International Conference on Computer Vision_ , pp. 5442–5451, October 2019.
  * Martinez et al. (2017) Julieta Martinez, Michael J Black, and Javier Romero.  On human motion prediction using recurrent neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2891–2900, 2017.
  * Nichol et al. (2021) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Pavlakos et al. (2019) Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3D hands, face, and body from a single image.  In _Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pp. 10975–10985, 2019.
  * Petrovich et al. (2021) Mathis Petrovich, Michael J. Black, and Gül Varol.  Action-conditioned 3D human motion synthesis with transformer VAE.  In _International Conference on Computer Vision (ICCV)_ , pp. 10985–10995, October 2021.
  * Petrovich et al. (2022) Mathis Petrovich, Michael J. Black, and Gül Varol.  TEMOS: Generating diverse human motions from textual descriptions.  In _European Conference on Computer Vision (ECCV)_ , 2022.
  * Plappert et al. (2016) Matthias Plappert, Christian Mandery, and Tamim Asfour.  The kit motion-language dataset.  _Big data_ , 4(4):236–252, 2016.
  * Raab et al. (2022) Sigal Raab, Inbal Leibovitch, Peizhuo Li, Kfir Aberman, Olga Sorkine-Hornung, and Daniel Cohen-Or.  Modi: Unconditional motion synthesis from diverse data.  _arXiv preprint arXiv:2206.08010_ , 2022.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pp. 8748–8763. PMLR, 2021.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , pp. 234–241. Springer, 2015.
  * Saharia et al. (2022a) Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In _ACM SIGGRAPH 2022 Conference Proceedings_ , pp. 1–10, 2022a.
  * Saharia et al. (2022b) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _arXiv preprint arXiv:2205.11487_ , 2022b.
  * Shi et al. (2020) Mingyi Shi, Kfir Aberman, Andreas Aristidou, Taku Komura, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen.  Motionet: 3d human motion reconstruction from monocular video with skeleton consistency.  _ACM Transactions on Graphics (TOG)_ , 40(1):1–15, 2020.
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International Conference on Machine Learning_ , pp. 2256–2265. PMLR, 2015.
  * Song et al. (2020a) Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song & Ermon (2020) Yang Song and Stefano Ermon.  Improved techniques for training score-based generative models.  _Advances in neural information processing systems_ , 33:12438–12448, 2020.
  * Song et al. (2020b) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * Tevet et al. (2022) Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or.  Motionclip: Exposing human motion generation to clip space.  _arXiv preprint arXiv:2203.08063_ , 2022.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Zhang et al. (2022) Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu.  Motiondiffuse: Text-driven human motion generation with diffusion model.  _arXiv preprint arXiv:2208.15001_ , 2022.