# Back to MLP: A Simple Baseline for Human Motion Prediction

Wen Guo*, Yuming Du*${\dagger}$, Xi Shen${\dagger}$, Vincent Lepetit, Xavier
Alameda-Pineda, and Francesc Moreno-Noguer $\star$ Equal
contribution.${\dagger}$ Corresponding author.W. Guo, X. Alameda-Pineda are
with INRIA Grenoble Rhône-Alpes and Université Grenoble Alpes, France. Wen is
also with Institut de Robòtica i Informàtica Industrial (CSIC-UPC), Barcelona,
08028, Spain. Email<EMAIL_ADDRESS>Du, V.
Lepetit are with LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, Marne-la-
vallee, France. Email: {yuming.du<EMAIL_ADDRESS>Shen is with
Tencent AI Lab, Shenzhen, China. Email<EMAIL_ADDRESS>Moreno-Noguer
is with Institut de Robòtica i Informàtica Industrial (CSIC-UPC), Barcelona,
08028, Spain. E-mail<EMAIL_ADDRESS>

###### Abstract

This paper tackles the problem of human motion prediction, consisting in
forecasting future body poses from historically observed sequences. Despite of
their performance, current state-of-the-art approaches rely on deep learning
architectures of arbitrary complexity, such as Recurrent Neural Networks
(RNN), Transformers or Graph Convolutional Networks (GCN), typically requiring
multiple training stages and more than 3 million of parameters. In this paper
we show that the performance of these approaches can be surpassed by a light-
weight and purely MLP architecture with only 0.14M parameters when
appropriately combined with several standard practices such as representing
the body pose with Discrete Cosine Transform (DCT), predicting residual
displacement of joints and optimizing velocity as an auxiliary loss.

An exhaustive evaluation on Human3.6M, AMASS and 3DPW datasets shows that our
method, which we dub siMLPe, consistently outperforms all other approaches. We
hope that our simple method could serve a strong baseline to the community and
allow re-thinking the problem of human motion prediction and whether current
benchmarks do really need intricate architectural designs. Our code is
available at https://github.com/dulucas/siMLPe.

###### Index Terms:

Human Motion Prediction, MLP, Simple Baseline.

##  1 Introduction

Given a sequence of 3D human poses, the task of human motion prediction aims
to predict the follow-up of the pose sequence. Forecasting future human motion
is of vital importance, and potential applications include preventing
accidents in autonomous driving [1], tracking people [2], or interacting with
a robot [3]

Due to the spatio-temporal nature of the human motion signal, the common trend
in the literature is to design models enable capture spatial and temporal
information. Traditional approaches mainly rely on hidden Markov models [4] or
Gaussian process latent variable models [5]. Although they perform well on
simple and periodic motion patterns, they fail dramatically when encountering
complex motions [6]. Recently, with the success of deep learning and neural
networks, various methods have been developed based on different types of
neural networks that are able to handle sequential data, including Recurrent
Neural Networks (RNN) [7], Graph Convolutional Networks (GCN) [6] and
Transformers [8]. However, these architectures are often complex and exploit
the aforementioned components to fuse the information across human joints and
time.

Figure 1: Comparison of parameter size and performance on Human3.6M dataset
[9]. we report MPJPE in mm at 1000 ms. Our method can achieve the lowest error
with significantly less parameters. 25]. We also show performances of two
simple baselines: ’Last Frame’ takes the last input frame as output, ’One FC’
uses only one single fully connected layer. These two naive methods could
already achieve comparable results than some classical approaches.

In this paper, we propose siMLPe, a simple yet effective network, which is
merely composed of multi layer perceptrons (MLPs) but able to encode the
variability of human motion. Specifically, our proposed network is illustrated
in Figure 2 and composed of a series of fully connected layers, layer
normalization [10] and spatial-temporal transpose operation. Even the commonly
used activation layers such as ReLU [11] are not necessary. Despite its
simplicity, siMLPe is able to achieve strong performance when appropriately
combined with standard practices, such as representing the body pose with
Discrete Cosine Transform (DCT), predicting residual displacement of joints
and optimizing velocity as an auxiliary loss.

Precisely, our approach siMLPe obtains state-of-the-art performance on several
standard benchmarks including Human3.6M [9], AMASS [12] and 3DPW [13]. siMLPe
is also light-weight and contains $20\times$ to $60\times$ less parameters
than competitive approaches. Figure 1 shows the comparison of Mean Per Joint
Position Error (MPJPE) at 1 000ms on Human3.6M [9] as well as the number of
parameters used in each approach, which demonstrates the effectiveness and
efficiency of our approach.

Interestingly, in extreme cases, leveraging the pose from the last frame
(‘Last-frame’) achieves long-term prediction close to Res-RNN [7] and using a
single FC layer (‘One-FC’) allows obtaining even better performance than Res-
RNN [7]. Note that Res-RNN [7] is a much more complex model based on RNN.
These observations suggest as a sequential prediction task, human motion
prediction can be tackled in a completely different way using directly simple
MLP. We expect our method could serve as a baseline for the future research
and help the community to rethink the task of human motion prediction. Our
implementation is available at https://github.com/dulucas/siMLPe.

In summary, our contributions are as follows:

  * •

We propose siMLPe, a simple yet effective baseline with only MLP layers for
human motion prediction, achieving state-of-the-art performance with far less
parameters than existing methods on multiple benchmarks like Human3.6M [9],
AMASS [12] and 3DPW [13] datasets.

  * •

We show that the human motion prediction can be modelled in a simple way
without explicitly fusing spatial and temporal information. As an extreme
example, a single FC layer can already achieve good performance.

##  2 Related Work

Human motion prediction is formulated as a sequence-to-sequence task, where
past observed motion is taken as input to predict the future motion sequence.
Traditional methods explore human motion prediction with nonlinear Markov
models [14], Gaussian Process dynamical models [15], and Restricted Boltzmann
Machine [16]. These approaches have shown to be effective to predict simple
motions and eventually struggle on complex and long-term motion prediction
[17]. In deep era, human motion prediction has achieved great success with the
use of deep networks, including Recurrent Neural Network(RNN) [17, 18, 7, 19,
20], Graph Convolutional Network(GCN) [6, 21, 22, 23, 24, 25, 26] and
Transformer [8], which are the main focus of this section.

###  2.1 RNN-based human motion prediction

Due to the inherent sequential structure of human motion, 3D human motion
prediction has been mostly addressed with recurrent models. Fragkiadaki et al.
[17] propose an encoder-decoder framework to embed human poses and an LSTM to
update the latent space and predict future motion. Jain et al. [18] manually
encodes the semantic similarity between different parts of the body and
forwards them via structural RNNs. However, these two methods suffer from
discontinuity and they only trained action-specific models, i.e. a single
model is trained for a specific action.

Martinez et al. [7] studied multi-actions instead of action-specific models
i.e. train one single model for multiple actions, which is in practice more
interesting and allows the network to exploit regularities across different
actions in large scale datasets. This is widely adopted by most of the motion
works until now. They also introduced a residual connection to model the
velocities instead of the absolute value to have more smooth predictions.

The above-mentioned methods also suffer from multiple inherent disadvantages
of RNN. First, as a sequential model, RNN is difficult for parallelization
during training and inference. Second, the memory constraints prevent RNN from
exploring information from farther frames. Some works alleviate this problem
by using RNN variants [19, 20], sliding windows [27, 28], convolutional models
[29, 30] or adversarial training [31].

###  2.2 GCN-based human motion prediction

To better encode the spatial connectivity of human joints, the most recent
works usually build the human pose as a graph and adopt Graph Convolutional
Networks(GCNs) [32, 33] for human motion prediction.

GCNs are first adopted in human motion prediction in Mao et al. [6]. They use
a stack of blocks consisting of GCNs, non linear activation and batch
normalization to encode the spatial dependencies, and leverage discrete cosine
transform (DCT) to encode temporal information. This work inspired most of the
GCN-based motion prediction methods in recent years. Based on [6], Mao et al.
[21] further improved the temporal encoding by cutting the past observations
into several sub-sequences, and adding an attention mechanism to find similar
previous motion sub-sequences in the past with the current observations. Thus
the future sequence is computed as a weighted sum of observed sub-sequences.
Then a GCN based predictor, exactly the same as that of Mao et al. [6], is
used to encode the spatial dependencies. Instead of using DCT to encode the
input sequence, [34] used a multi-scale temporal input embedding, by applying
various size convolutional for different input sizes to have different
receptive fields in the temporal domain. Ma et al. [23] proposes two variants
of GCN to extract spatial and temporal features. They build a multi-stage
structure where each stage contains an encoder and a decoder, and during the
training, the model is trained with intermediate supervision to learn to
progressively refine the prediction. Besides, [24, 25, 26] extend the graph of
human pose to multi-scale version across the abstraction levels of human pose.

###  2.3 Attention-based human motion prediction

With the development of transformer [35], some works [21, 8, 36] tried to deal
with this task with attention mechanism. Beside of [21] which adopted
attention to find temporal relations, [8] use attentions to map not only the
temporal dependencies but also the pairwise relation of joints by an
architecture which combines spatial-attention and temporal-attention in
parallel. Moreover, [36] used a transformer-based architecture along with a
progressive-decoding strategy to predict the DCT coefficients of the target
joints progressively based on the kinematic tree. In order to guide the
predictions, they also built a memory-based dictionary to preserve the global
motion patterns in training data.

With the development of human motion prediction in recent years, the
RNN/GCN/transformer based architectures are well explored and the results have
been significantly improved. Though these methods are shown to be effective,
the architectures are becoming more and more complicated and difficult to
train. In this paper, we stick to simple architectures and propose a MLP-based
network. We hope that our simple method would serve a baseline and let the
community rethink the problem of human motion prediction.

Figure 2: Overview of our approach siMLPe for human motion prediction. FC
denotes a fully connected layer, Layer Norm denotes a layer normalization [10]
operation. DCT and IDCT represent the discrete cosine transformation and
inverse discrete cosine transformations respectively. The MLP blocks in gray
(composed of FC and Layer Norm) are repeated n times.

##  3 Our approach : siMLPe

Given a sequence of 3D human poses in the past, our goal is to predict the
future 3D human poses. Our approach is illustrated in Figure 2. We first
present how to pre-process motion data in Section 3.1. The components of the
network are detailed in Section 3.2. In Section 3.3, we detail the losses we
use for training.

We denote the observed 3D human poses as
$\mathbf{x}_{1:T}=[x_{1}^{\top},..,x_{T}^{\top}]^{\top}\in\mathbf{R}^{T\times
C}$, consisting of $T$ consecutive human poses, where the pose at the $t$-th
frame $x_{t}$ is represented by a $C$-dimensional feature, _i.e._
$x_{t}\in\mathbf{R}^{C}$. Our goal is to predict the future $N$ motion frames
$\mathbf{x}_{T+1:T+N}=[x_{T+1}^{\top},..,x_{T+N}^{\top}]^{\top}\in\mathbf{R}^{N\times
C}$. In this work, we follow previous works [7, 6, 21, 23] and adopt the 3D
coordinates of the joints as our motion representation.

TABLE I: Results on Human3.6M [9]. We report MPJPE error in mm. ${\dagger}$
indicates that the result is taken from Hisrep [21], $\star$ indicated that
the result is taken from ST-DGCN [23]. We also show results of two simple
baselines: ’Last Frame’ takes the last input frame as output, ’One FC’ uses
only one single fully connected layer. These two implementations could already
achieve comparable results with early methods like [7].

| MPJPE (mm) $\downarrow$ | $\\#$ Param.(M) $\downarrow$
---|---|---
Time (ms) | 80 | 160 | 320 | 400 | 560 | 720 | 880 | 1000
Last Frame | 23.8 | 44.4 | 76.1 | 88.2 | 107.4 | 121.6 | 131.6 | 136.6 | 0
One FC | 14.0 | 33.2 | 68.0 | 81.5 | 101.7 | 115.1 | 124.8 | 130.0 | 0.003
Res-RNN ${\dagger}$ [7] | 25.0 | 46.2 | 77.0 | 88.3 | 106.3 | 119.4 | 130.0 | 136.6 | 3.44
convSeq2Seq ${\dagger}$ [30] | 16.6 | 33.3 | 61.4 | 72.7 | 90.7 | 104.7 | 116.7 | 124.2 | 15.58
LTD-10-10 ${\dagger}$ [6] | 11.2 | 23.4 | 47.9 | 58.9 | 78.3 | 93.3 | 106.0 | 114.0 | 2.56
Hisrep ${\dagger}$ [21] | 10.4 | 22.6 | 47.1 | 58.3 | 77.3 | 91.8 | 104.1 | 112.1 | 3.24
MSR-GCN $\star$ [24] | 11.3 | 24.3 | 50.8 | 61.9 | 80.0 | - | - | 112.9 | 6.30
ST-DGCN-10-25 $\star$ [23] | 10.6 | 23.1 | 47.1 | 57.9 | 76.3 | 90.7 | 102.4 | 109.7 | 3.80
siMLPe(Ours) | 9.6 | 21.9 | 46.5 | 57.5 | 75.8 | 90.1 | 101.8 | 109.5 | 0.14

TABLE II: Results on AMASS-BMLrub [12] and 3DPW [13]. We report MPJPE error in
mm. The model is trained on AMASS dataset. State-of-the-art results are taken
from Hisrep [21].

| AMASS | 3DPW
---|---|---
Time (ms) | 80 | 160 | 320 | 400 | 560 | 720 | 880 | 1000 | 80 | 160 | 320 | 400 | 560 | 720 | 880 | 1000
convSeq2Seq [30] | 20.6 | 36.9 | 59.7 | 67.6 | 79.0 | 87.0 | 91.5 | 93.5 | 18.8 | 32.9 | 52.0 | 58.8 | 69.4 | 77.0 | 83.6 | 87.8
LTD-10-10 [6] | 10.3 | 19.3 | 36.6 | 44.6 | 61.5 | 75.9 | 86.2 | 91.2 | 12.0 | 22.0 | 38.9 | 46.2 | 59.1 | 69.1 | 76.5 | 81.1
LTD-10-25 [6] | 11.0 | 20.7 | 37.8 | 45.3 | 57.2 | 65.7 | 71.3 | 75.2 | 12.6 | 23.2 | 39.7 | 46.6 | 57.9 | 65.8 | 71.5 | 75.5
Hisrep [21] | 11.3 | 20.7 | 35.7 | 42.0 | 51.7 | 58.6 | 63.4 | 67.2 | 12.6 | 23.1 | 39.0 | 45.4 | 56.0 | 63.6 | 69.7 | 73.7
siMLPe(Ours) | 10.8 | 19.6 | 34.3 | 40.5 | 50.5 | 57.3 | 62.4 | 65.7 | 12.1 | 22.1 | 38.1 | 44.5 | 54.9 | 62.4 | 68.2 | 72.2

###  3.1 Discrete Cosine Transform (DCT)

We use the Discrete Cosine Transform (DCT) to encode temporal information,
which is proven to be beneficial for human motion prediction [6, 21, 23]. More
precisely, given an input motion sequence of $T$ frames, the DCT matrix
$\mathbf{D}\in\mathbf{R}^{T\times T}$ can be calculated as:

$\displaystyle\mathbf{D}_{i,j}=\sqrt{\frac{2}{T}}\frac{1}{\sqrt{1+\delta_{i,0}}}\cos\left(\frac{\pi}{2T}(2j+1)i\right)\>,$
(1)

where $\delta_{i,j}$ denotes the  _Kronecker_ delta function:

$\displaystyle\delta_{i,j}=\begin{cases}1&\text{if}\ i=j\\\ 0&\text{if}\ i\neq
j\>.\end{cases}$ (2)

The transformed input is
$\mathcal{D}(\mathbf{x}_{1:T})=\mathbf{D}\mathbf{x}_{1:T}$. For the output, we
apply the Inverse Discrete Cosine Transform (IDCT) to transform the output of
the network to the original pose representation, denoted as $\mathcal{D}^{-1}$
and the inverse of $\mathbf{D}$.

###  3.2 Network Architecture

The architecture of our network is shown in Figure 2. Our network only
contains three components, the fully connected layers, the transpose operation
and layer normalization [10]. For the all the fully connected layers, their
input dimension is equal to their output dimension. In this paper, we call a
block containing one fully connected layer and one layer normalization [10]
with skip connection a MLP block, as shown in the Figure 2. After the DCT
transformation, a motion sequence is first fed to a fully connected layer and
a transpose layer, then the output feature is fed to $n$ consecutive MLP
blocks. In the end, the extracted feature is transposed back to the original
dimension and fed to a fully connected layer, the predicted motion is obtained
by applying the IDCT to the output of the last fully connected layer.

###  3.3 Losses

In practice, instead of predicting the absolute coordinates of joints in each
frame, we predict the difference between the future frames and the last frame
of the input motion sequence. Therefore, the output of the network is actually
$x_{T+t}-x_{T}$.

#### Objective function.

Our objective function $\mathcal{L}$ includes two terms $\mathcal{L}_{re}$ and
$\mathcal{L}_{v}$,

$\displaystyle\mathcal{L}$ $\displaystyle=\mathcal{L}_{re}+\mathcal{L}_{v}\>.$
(3)

$\mathcal{L}_{re}$ aims at minimizing the L2-norm between the predicted motion
$\mathbf{x^{\prime}}_{T+1:T+N}$ and ground-truth one $\mathbf{x}_{T+1:T+N}$:

$\displaystyle\mathcal{L}_{re}$
$\displaystyle=\mathcal{L}_{2}(\mathbf{x^{\prime}}_{T+1:T+N},\mathbf{x}_{T+1:T+N})\>.$
(4)

$\mathcal{L}_{v}$ is a loss on velocity and aims at minimizing the difference
between the velocity of the predicted motion $\mathbf{v^{\prime}}_{T+1:T+N}$
and the ground truth one $\mathbf{v}_{T+1:T+N}$:

$\displaystyle\mathcal{L}_{v}$
$\displaystyle=\mathcal{L}_{2}(\mathbf{v^{\prime}}_{T+1:T+N},\mathbf{v}_{T+1:T+N})\>.$
(5)

where
$\mathbf{v}_{T+1:T+N}=[v_{T+1}^{\top},..,v_{T+N}^{\top}]^{\top}\in\mathbf{R}^{N\times
C}$, $v_{t}$ represents the velocity at frame $t$ and is calculated as
$v_{t}=x_{t+1}-x_{t}$. We provide a full analysis on the loss terms in Section
4.5.

##  4 Experiments

In this section, we present our experimental results. We first introduce
datasets and evaluation metric in Section 4.1. Implementation details are
provided in Section 4.2. We then present quantitative and qualitative results
in Section 4.3. Finally, exhaustive analysis on the architecture, losses are
detailed in Section 4.5.

###  4.1 Datasets and evaluation metric

#### Human3.6M [9] dataset.

Human3.6M contains 7 actors performing 15 actions, 32 joints are labeled for
each human. We use the same test protocols of [21], and use S5 as the test
set, S11 as the validation set and the others as training set. Previous works
use different testing sampling strategies, including 8 samples per action [7,
6], 256 samples per action [21] or all samples in the test set [24]. We take
256 samples per action for testing, and evaluate on 22 joints as in [7, 6, 21,
23].

#### AMASS [12] dataset.

AMASS is a collection of multiple Mocap datasets [37, 38, 39, 40, 41, 42, 43,
44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54] scandalized by SMPL [55]
parameterization. We follow [21] to use BMLrub [41] as test set, and split the
rest of AMASS [12] dataset into training and validation set. The model is
tested on 18 joints for evaluation as in [21].

#### 3DPW [13] dataset.

3DPW is a dataset including indoor and outdoor scenes. Each pose is
represented by 26 joints. We follow [21] to evaluate on 3DPW [13] on 18 joints
using the model trained on AMASS [12] to show the generalisation of our
method.

#### Evaluation metric.

We report Mean Per Joint Position Error (MPJPE) on 3D joint coordinates, which
is the most widely used metric for evaluating 3D pose errors. This metric
calculates the average l2-norm across different joints between the prediction
and ground-truth. Similar with previous works [6, 21, 24, 23], we ignore the
global rotation and translation of the poses and keep the sampling rate as 25
frame per second (FPS) for all datasets.

###  4.2 Implementation details

TABLE III: Analysis of the number of MLP blocks on Human3.6M [9]. We report
number of parameters in M as well as MPJPE error in mm.

Nb. Blocks | $\\#$ Param.(M) $\downarrow$ | MPJPE (mm) $\downarrow$
---|---|---
80 | 160 | 320 | 400 | 560 | 720 | 880 | 1000
1 | 0.012 | 12.7 | 28.5 | 59.7 | 72.1 | 93.6 | 107.0 | 116.8 | 123.6
2 | 0.014 | 10.9 | 24.9 | 52.3 | 64.0 | 83.2 | 97.3 | 108.4 | 115.4
6 | 0.025 | 10.2 | 23.1 | 48.8 | 60.1 | 79.0 | 93.3 | 105.1 | 112.6
12 | 0.041 | 9.9 | 22.4 | 47.2 | 58.3 | 77.1 | 91.5 | 103.3 | 110.9
24 | 0.073 | 9.7 | 22.0 | 46.8 | 57.7 | 76.4 | 90.8 | 102.6 | 110.3
48 (Ours) | 0.138 | 9.6 | 21.8 | 46.4 | 57.4 | 76.0 | 90.2 | 101.9 | 109.5
64 | 0.180 | 9.6 | 21.8 | 46.5 | 57.5 | 76.0 | 90.1 | 101.9 | 109.7
96 | 0.266 | 9.7 | 21.9 | 46.7 | 57.8 | 76.3 | 90.5 | 102.1 | 109.8

TABLE IV: Analysis of the MLP block on Human3.6M [9]. We report MPJPE error in
mm.

Transpose | Layer | MPJPE (mm) $\downarrow$
---|---|---
Norm | 80 | 160 | 320 | 400 | 560 | 720 | 880 | 1000
|  | 23.7 | 44.0 | 75.5 | 87.6 | 106.3 | 120.4 | 130.5 | 135.6
| ✓ | 23.8 | 43.0 | 73.4 | 85.2 | 102.0 | 116.3 | 125.3 | 131.9
✓ |  | 12.7 | 29.0 | 62.3 | 76.2 | 97.4 | 111.6 | 121.6 | 127.3
✓ | ✓ | 9.6 | 21.8 | 46.4 | 57.4 | 76.0 | 90.2 | 101.9 | 109.5

TABLE V: Ablation of different losses on Human3.6M [9]. We report MPJPE error
in mm.

$\mathcal{L}_{re}$ | $\mathcal{L}_{v}$ | MPJPE (mm) $\downarrow$
---|---|---
80 | 160 | 320 | 400 | 560 | 720 | 880 | 1000
✓ |  | 9.6 | 21.8 | 46.5 | 57.5 | 76.7 | 91.5 | 103.5 | 111.3
✓ | ✓ | 9.6 | 21.9 | 46.5 | 57.5 | 75.8 | 90.1 | 101.8 | 109.5

In practice, we set input length $T=50$ and output length $N=10$. During
training, we take 50 frames as input and supervise only the first 10 frames of
the output. During testing, we apply our model in an auto-regressive manner to
generate motion for longer frames. The feature dimension $C=3\times K$, where
$K$ is the number of tested joints and varies across different datasets.

To train our network, we set batch size to 256 and use Adam [56] optimizer,
the memory consumed by our network is about $2GB$ during the training. We
repeat each experiment 5 times and take the average of the results. All our
experiments are conducted using Pytorch [57] framework on a single NVIDIA RTX
2080Ti graphics card. We train our network on Human3.6M [9] dataset for 35k
iterations, the learning rate starts from 0.0003 at the beginning and drops to
0.00001 after 30k steps. The training takes $\sim$30 minutes. And we train our
network on AMASS [12] dataset for 115k iterations, the learning rate starts
from 0.0003 at the beginning and drops to 0.00001 after 100k steps. The entire
training finishes in $\sim$2 hours.

###  4.3 Comparison to state-of-the-art approaches

In this section, we compare our approach to existent state-of-the-art methods
on different datasets. For each dataset, we report MPJPE in mm on different
prediction time steps up to 1000ms.

#### Human3.6M [9] dataset.

We compare our method with other state-of-the-art methods on Human3.6M
dataset, as shown in Table I. Our method outperforms all previous methods on
every frame with much less number of parameters.
As explained in Section 4.1, different methods have taken different test
sampling strategies, and we choose to test on 22 joints with 256 samples per
action, following [21]. So to make fair comparison, we evaluate all the
methods using the same testing protocol. Previous works usually report short-
term(0-500ms) and long-term(500-1000ms) predictions separately, and some works
[23] report short-/long- term results using two different trained models. In
our tables, all the results from 0 to 1000ms are generated by a single model,
and for [23] we report results of their model which achieve the best
performance on long term prediction.

In addition, we show two experimental results on the Human3.6m dataset in
Table I: ‘Last-frame’ takes the last input frame and repeat it $N$ times as
output, ‘One FC’ uses only one single fully connected layer trained on
Human3.6m dataset. These two simple implementations could already achieve
comparable results with early methods like [7].

#### AMASS [12] and 3DPW [13] dataset.

In Table II, we report performance of the model trained on AMASS dataset and
tested on AMASS-BMLrub and 3DPW dataset, following the setting of [21].
Different from Human3.6m dataset where the training and testing data are same
actions performed by different actors, the difference between training/testing
data under this protocol is much bigger, which challenges the generalization
of the method. We can see that our approach achieves competitive performances
on short term prediction ($80ms$ and $160ms$) and perform consistently better
on long-term prediction ($>160ms$). More importantly, our model is much
smaller and only contains $\sim 4\%$ parameters compared to Hisrep [21].

###  4.4 Qualitative result

Figure 3 shows some qualitative results of our methods comparing with other
state-of-the-art methods, on Human3.6M dataset. The prediction of our methods
is closer to the ground truth.

###  4.5 Ablation study

In this section, we ablate different components of our approach on Human3.6M
dataset.

#### Number of blocks.

We ablate the number of blocks $n$ in Table III. Our proposed architecture can
already achieve reasonable performance using only 2 blocks, with parameter
size of 0.014M. And the network achieves its best performance with 48 blocks.

#### Network Architecture.

In Table IV, we ablate the components of our network. As shown in the table,
the transpose operation and layer normalization [10] is of vital importance to
our network. Without the transpose operation, the network can only operate on
spatial dimension of the motion sequence without merging any information
across different frames, which leads to degraded results. This also reveals
the fact that merging temporal information across different frames is the key
of human motion prediction.

#### Loss.

In Table V we ablate the importance of different loss terms used during
training. As shown in the table, with the help of velocity loss
$\mathcal{L}_{v}$, the network can achieve better performance on long term
prediction while maintaining the same performance on short term.

Figure 3: Some qualitative results of our method siMLPe. The skeletons with
light color are ground-truth, while those with dark color are predicted
motions. Our prediction results are close to the ground-truth.

##  5 Conclusion

In this paper, we present siMLPe, a simple-yet-effective network for human
motion prediction. Our network is composed of only fully connected layer,
layer normalization and transpose operation. While using much less parameters,
our network can achieve state-of-the-art performance on various benchmarks. We
release our code and hope the simplicity of our method could help to community
to rethink this task.

## Acknowledgments

This project has received funding from the CHIST-ERA IPALM project and was
also supported by ANR-3IA MIAI (ANR-19-P3IA0003), ANR-JCJC ML3RI
(ANR-19-CE33-0008-01), H2020 SPRING (funded by EC under GA $\\#871245$), by
the Spanish government with the project MoHuCo PID2020-120049RB-I00 and by an
Amazon Research Award.

## References

  * [1] B. Paden, M. Čáp, S. Z. Yong, D. Yershov, and E. Frazzoli, “A survey of motion planning and control techniques for self-driving urban vehicles,” _Transactions on intelligent vehicles_ , 2016.
  * [2] H. Gong, J. Sim, M. Likhachev, and J. Shi, “Multi-hypothesis motion planning for visual object tracking,” in _ICCV_ , 2011.
  * [3] H. S. Koppula and A. Saxena, “Anticipating human activities for reactive robotic response.” in _IROS_ , 2013.
  * [4] M. Brand and A. Hertzmann, “Style machines,” in _Computer Graphics and Interactive Techniques_ , 2000.
  * [5] J. M. Wang, D. J. Fleet, and A. Hertzmann, “Gaussian process dynamical models for human motion,” _PAMI_ , 2007.
  * [6] W. Mao, M. Liu, M. Salzmann, and H. Li, “Learning trajectory dependencies for human motion prediction,” in _ICCV_ , 2019.
  * [7] J. Martinez, M. J. Black, and J. Romero, “On human motion prediction using recurrent neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2891–2900.
  * [8] E. Aksan, M. Kaufmann, P. Cao, and O. Hilliges, “A spatio-temporal transformer for 3d human motion prediction,” in _3DV_ , 2021.
  * [9] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, “Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments,” _PAMI_ , 2013.
  * [10] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv_ , 2016\.
  * [11] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in _Icml_ , 2010.
  * [12] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black, “Amass: Archive of motion capture as surface shapes,” in _ICCV_ , 2019.
  * [13] T. von Marcard, R. Henschel, M. Black, B. Rosenhahn, and G. Pons-Moll, “Recovering accurate 3d human pose in the wild using imus and a moving camera,” in _European Conference on Computer Vision (ECCV)_ , sep 2018.
  * [14] A. M. Lehrmann, P. V. Gehler, and S. Nowozin, “Efficient nonlinear markov models for human motion,” in _CVPR_ , 2014.
  * [15] J. M. Wang, D. J. Fleet, and A. Hertzmann, “Gaussian process dynamical models,” in _NeurIPS_ , 2005.
  * [16] G. W. Taylor, G. E. Hinton, and S. T. Roweis, “Modeling human motion using binary latent variables,” in _NeurIPS_ , 2007.
  * [17] K. Fragkiadaki, S. Levine, P. Felsen, and J. Malik, “Recurrent network models for human dynamics,” in _ICCV_ , 2015.
  * [18] A. Jain, A. R. Zamir, S. Savarese, and A. Saxena, “Structural-rnn: Deep learning on spatio-temporal graphs,” in _CVPR_ , 2016.
  * [19] Z. Liu, S. Wu, S. Jin, Q. Liu, S. Lu, R. Zimmermann, and L. Cheng, “Towards natural and accurate future motion prediction of humans and animals,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 004–10 012.
  * [20] H.-k. Chiu, E. Adeli, B. Wang, D.-A. Huang, and J. C. Niebles, “Action-agnostic human pose forecasting,” in _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2019, pp. 1423–1432.
  * [21] W. Mao, M. Liu, and M. Salzmann, “History repeats itself: Human motion prediction via motion attention,” in _European Conference on Computer Vision_. Springer, 2020, pp. 474–489.
  * [22] W. Guo, X. BIE, X. Alameda-Pineda, and F. Moreno-Noguer, “Multi-person extreme motion prediction,” _CVPR_ , 2022.
  * [23] T. Ma, Y. Nie, C. Long, Q. Zhang, and G. Li, “Progressively generating better initial guesses towards next stages for high-quality human motion prediction,” in _CVPR_ , 2022.
  * [24] L. Dang, Y. Nie, C. Long, Q. Zhang, and G. Li, “Msr-gcn: Multi-scale residual graph convolution networks for human motion prediction,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2021, pp. 11 467–11 476.
  * [25] M. Li, S. Chen, Y. Zhao, Y. Zhang, Y. Wang, and Q. Tian, “Dynamic multiscale graph neural networks for 3d skeleton based human motion prediction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 214–223.
  * [26] M. Li, S. Chen, X. Chen, Y. Zhang, Y. Wang, and Q. Tian, “Symbiotic graph neural networks for 3d skeleton-based human action recognition and motion prediction,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [27] J. Butepage, M. J. Black, D. Kragic, and H. Kjellstrom, “Deep representation learning for human motion prediction and classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 6158–6166.
  * [28] J. Bütepage, H. Kjellström, and D. Kragic, “Anticipating many futures: Online human motion prediction and generation for human-robot interaction,” in _2018 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2018, pp. 4563–4570.
  * [29] A. Hernandez, J. Gall, and F. Moreno-Noguer, “Human motion prediction via spatio-temporal inpainting,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 7134–7143.
  * [30] C. Li, Z. Zhang, W. S. Lee, and G. H. Lee, “Convolutional sequence to sequence model for human dynamics,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5226–5234.
  * [31] L.-Y. Gui, Y.-X. Wang, X. Liang, and J. M. Moura, “Adversarial geometry-aware human motion prediction,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 786–803.
  * [32] A. Sperduti and A. Starita, “Supervised neural networks for the classification of structures,” _Transactions on Neural Networks_ , 1997.
  * [33] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in _ICLR_ , 2017.
  * [34] T. Lebailly, S. Kiciroglu, M. Salzmann, P. Fua, and W. Wang, “Motion prediction using temporal inception module,” in _Proceedings of the Asian Conference on Computer Vision_ , 2020.
  * [35] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , 2017.
  * [36] Y. Cai, L. Huang, Y. Wang, T.-J. Cham, J. Cai, J. Yuan, J. Liu, X. Yang, Y. Zhu, X. Shen _et al._ , “Learning progressive joint propagation for human motion prediction,” in _European Conference on Computer Vision_. Springer, 2020, pp. 226–242.
  * [37] Advanced Computing Center for the Arts and Design, “ACCAD MoCap Dataset.” [Online]. Available: https://accad.osu.edu/research/motion-lab/mocap-system-and-data
  * [38] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black, “AMASS: Archive of motion capture as surface shapes,” in _ICCV_ , 2019.
  * [39] A. Aristidou, A. Shamir, and Y. Chrysanthou, “Digital dance ethnography: Organizing large dance collections,” _J. Comput. Cult. Herit._ , vol. 12, no. 4, Nov. 2019. [Online]. Available: https://doi.org/10.1145/3344383
  * [40] B. M. Lab, “BMLhandball Motion Capture Database.” [Online]. Available: https://www.biomotionlab.ca//
  * [41] N. F. Troje, “Decomposing biological motion: A framework for analysis and synthesis of human gait patterns,” _Journal of Vision_ , vol. 2, no. 5, pp. 2–2, Sep. 2002.
  * [42] Carnegie Mellon University, “CMU MoCap Dataset.” [Online]. Available: http://mocap.cs.cmu.edu
  * [43] F. Bogo, J. Romero, G. Pons-Moll, and M. J. Black, “Dynamic FAUST: Registering human bodies in motion,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jul. 2017, pp. 5573–5582.
  * [44] E. J. C. Ltd., “Eyes Japan MoCap Dataset.” [Online]. Available: http://mocapdata.com
  * [45] S. Ghorbani, K. Mahdaviani, A. Thaler, K. Kording, D. J. Cook, G. Blohm, and N. F. Troje, “MoVi: A large multipurpose motion and video dataset,” 2020.
  * [46] A. Chatzitofis, L. Saroglou, P. Boutis, P. Drakoulis, N. Zioulis, S. Subramanyam, B. Kevelham, C. Charbonnier, P. Cesar, D. Zarpalas _et al._ , “Human4d: A human-centric multimodal dataset for motions and immersive media,” _IEEE Access_ , vol. 8, pp. 176 241–176 262, 2020.
  * [47] L. Sigal, A. Balan, and M. J. Black, “HumanEva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion,” _International Journal of Computer Vision_ , vol. 87, no. 4, pp. 4–27, Mar. 2010.
  * [48] C. Mandery, Ö. Terlemez, M. Do, N. Vahrenkamp, and T. Asfour, “The kit whole-body human motion database,” in _ICAR_ , 2015.
  * [49] M. Loper, N. Mahmood, and M. J. Black, “MoSh: Motion and Shape Capture from Sparse Markers,” _ACM Trans. Graph._ , vol. 33, no. 6, Nov. 2014. [Online]. Available: https://doi.org/10.1145/2661229.2661273
  * [50] M. Müller, T. Röder, M. Clausen, B. Eberhardt, B. Krüger, and A. Weber, “Documentation mocap database HDM05,” Universität Bonn, Tech. Rep. CG-2007-2, Jun. 2007.
  * [51] I. Akhter and M. J. Black, “Pose-conditioned joint angle limits for 3D human pose reconstruction,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jun. 2015, pp. 1446–1455.
  * [52] S. F. University and N. U. of Singapore, “SFU Motion Capture Database.” [Online]. Available: http://mocap.cs.sfu.ca/
  * [53] L. Hoyet, K. Ryall, R. McDonnell, and C. O’Sullivan, “Sleight of hand: Perception of finger motion from reduced marker sets,” in _Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games_ , ser. I3D ’12, New York, NY, USA, 2012, p. 79–86. [Online]. Available: https://doi.org/10.1145/2159616.2159630
  * [54] M. Trumble, A. Gilbert, C. Malleson, A. Hilton, and J. Collomosse, “Total capture: 3d human pose estimation fusing video and inertial sensors,” in _2017 British Machine Vision Conference (BMVC)_ , 2017.
  * [55] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, “Smpl: A skinned multi-person linear model,” _ACM transactions on graphics (TOG)_ , vol. 34, no. 6, pp. 1–16, 2015.
  * [56] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [57] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in neural information processing systems_ , vol. 32, 2019.


11institutetext: 1 Univ. Lyon, Univ. Lyon 1, CNRS/IN2P3, IP2I, F-69622,
Villeurbanne, France
11email<EMAIL_ADDRESS>
2 Institut Néel, Université Grenoble-Alpes, CNRS/INP, 25 rue des Martyrs,
Grenoble, France
3 LPSC, Université Grenoble-Alpes, CNRS/IN2P3, 53, rue des Martyrs, Grenoble,
France
4 C2N, CNRS, Univ. Paris-Sud, Univ. Paris-Saclay, 91120 Palaiseau, France
5 CRYOHEMT S.A.S 91400 Orsay, France

# HEMT-based 1K front-end electronics for the heat and ionization Ge CryoCube
of the future Ricochet CE$\nu$NS experiment

G. Baulieu1 J. Billard1 G. Bres2 J-L Bret2 D. Chaize1 J. Colas1 Q. Dong5 O.
Exshaw2 C. Guerin1 S. Ferriol1 J-B Filippini1 M. De Jesus1 Y. Jin4 A.
Juillard1 J. Lamblin3 H. Lattaud1 J. Minet2 D. Misiak1 A. Monfardini2 F.
Rarbi3 T. Salagnac1 L. Vagneron1 and the Ricochet Collaboration

###### Abstract

The Ricochet reactor neutrino observatory is planned to be installed at the
Laue Langevin Institute (ILL) starting mid-2022. Its scientific goal is to
perform a low-energy and high precision measurement of the coherent elastic
neutrino-nucleus scattering (CE$\nu$NS) spectrum in order to explore exotic
physics scenarios. Ricochet will host two cryogenic detector arrays: the
CryoCube (Ge target) and the Q-array (Zn target), operated at 10 mK. The 1 kg
Ge CryoCube will consist of 27 Ge crystals instrumented with NTD-Ge thermal
sensors and charge collection electrodes for a simultaneous heat and
ionization readout to reject the electromagnetic backgrounds (gamma, beta,
x-rays). We present the status of its front-end electronics. The first stage
of amplification is made of High Electron Mobility Transistor (HEMT) developed
by CNRS/C2N laboratory, optimized to achieve ultra-low noise performance at 1K
with a dissipation as low as 15 $\mu$W per channel. Our noise model predicts
that 10 eV heat and 20 eVee RMS baseline resolutions are feasible with a high
dynamic range for the deposited energy (up to 10 MeV) thanks to loop
amplification schemes. Such resolutions are mandatory to have a high
discrimination power between nuclear and electron recoils at the lowest
energies.

###### Keywords:

HEMT, amplifier, Cryoelectronics, CE$\nu$NS, cryogenic detectors

††journal: Journal of Low Temperature Physics

##  1 Introduction

High-purity cryogenic semiconductor detectors with simultaneous readout of
phonon and ionization signals have been developed over the past three decades.
These developments were initially motivated by the direct search for Dark
Matter in the form of Weakly Interacting Massive Particles (WIMPs) and more
recently by the search of new physics with precise measurement of the
CE$\nu$NS spectrum: in both cases the signal is a nuclear recoil while the
background are electron recoils induced by $\alpha$, $\beta$ and $\gamma$
radioactivity and heat-only events.

The COHERENT collaboration reported the first observation of CE$\nu$NS from
$\sim$ 30 MeV neutrinos induced by the SNS spallation source Coherent . At
this high energy the scattering is not fully coherent and the influence of the
new physics is expected to be more visible with neutrinos of few MeV. In
germanium, the nuclear recoil energy spectrum for 5 MeV neutrinos is similar
to that of a 2.7 GeV WIMP, this explains the important synergy between the
development of low mass dark matter and CE$\nu$NS detectors. The cryogenic
Dark Matter detector community has therefore started to work on projects such
as Ricochet, NUCLEUS and MINER Ricochet ; Nucleus ; Miner  to measure
CE$\nu$NS from few MeV neutrinos produced by nuclear reactors.

Ricochet will be installed at the Laue Langevin Institute ILL  in Grenoble
(France). Two cryogenic detector arrays, the CryoCube (Ge target) and the
Q-array (Zn target) will be running in a dilution fridge at 8.8 meter away
from the 58 MW nuclear reactor, shielded by 20 cm of Pb and 35 cm of PE, and
surrounded by an active plastic scintillator muon veto. The site provides an
overburden of 15 m.w.e. More details on the Ricochet setup are given in
Ricochet , the CryoCube is described in CryoCube . With a $\sim$ 1 kg payload,
a total of 13 CE$\nu$NS interactions are expected per day. The Ge detectors
will be of two types: planar detectors with 2 charge collection electrodes
(top and bottom) and Full Interditized Detectors adapted from the FID800
detectors developed by the EDELWEISS collaboration EDW_JINST  and fitted by 4
electrodes (2 on the top, 2 on the bottom). All detectors will be equipped
with 1 Ge-NTD heat sensor. One of the main challenge of Ricochet is to keep an
active discrimination for nuclear recoils at thresholds as low as 50 eVnr. As
the actual background at such low energy is still unknown and all cryogenic
experiments exhibit huge excesses at low energy EXCESS  this approach is an
important risk mitigation. Due to the low ionization yield for nuclear
recoils, the detection threshold is driven by the heat channel while the
discrimination threshold is limited by the ionization channel resolution. A
baseline resolution of $\sigma$ = 20 eVee RMS (eV-electron equivalent,
ionization energy produced by electronic recoils) is needed to meet the goal.
This is one order of magnitude below the best resolution reached by cryogenic
Ge detectors with Si-JFET EDW_JINST . The heat channel baseline resolution
goal is 10 eV (RMS) which is less than a factor 2 of the best achieved
resolution on such small Ge detector by the EDELWEISS collaboration EDW_Surf .

In previous work HEMT_LTD18  we have shown that HEMTs developed by the
CNRS/C2N laboratory HEMT_YongAPL ; HEMT_ICSICT , and now commercially
available at the CryoHEMT company CryoHEMT , could be used as high impedance
input and could meet the resolution goals with a total input (detector +
cabling) capacitance of 20 pF. The noise measurements were made at 4 K with a
dissipation of 100 $\mu$W. The extrapolation to the 150 channels needed for
the CryoCube is not straightforward. In this paper, we show that these HEMTs
keep their performance at 1 K with power dissipation as low as 15 $\mu$W and
exhibit a good reproducibility over about 100 HEMT tested. As a result the
cold electronics could be placed at 1 K, very close to the detector stage to
mitigate the cabling capacitance. In addition we show that precise gain and
high linearity are obtained thanks to loop amplifiers, first measurements at
low temperature are presented.

##  2 HEMT performance and reproducibility at 4K and 1K

Figure 1: Top Left: Ids vs Vds characteristics at 4K and 1K for a Cgs=100 pF
HEMT device. From top to bottom a $\delta$Vgs = -1 mV step is applied between
each curve for the 1K data, only 4 curves are shown for the 4K data (thick
blue) to highlight the good match with 1K curves. Top Right and Bottom Left:
Extracted transconductances gm and output conductances gd. 1K and 4K (thick
blue) values are compatible within the measurement precision. Bottom Right:
Voltage gain AV. 1K and 4K data (thick blue) are compatible (Color figure
online.)

The CNRS/C2N laboratory is developing HEMTs of various geometries, with input
capacitance Cgs varying from 2 pF to 230 pF. Ids vs Vds characteristics for
various Vgs have been first measured at both 4 K and 1 K up to a saturation
current of $\sim$1 mA and from this data we have extracted the
transconductance $g_{m}=\delta I_{ds}/\delta V_{gs}$ and the output
conductance $g_{d}=\delta I_{ds}/\delta V_{ds}$. The voltage gain AV is
measured in a standard common source amplifier configuration with a 1
k$\Omega$ load resistance on the drain HEMT_LTD18 . Typical results are given
in Fig. 1 for a Cgs = 100 pF device. The 1 K and 4 K characteristics are
similar. The saturation decreases from 1 mA to 0.1 mA by varying Vgs from -75
mV to -115 mV. The Ids vs Vds characteristics are shifted by only 1mV. The
extracted gm and gd at 1 K and 4 K are compatible within the precision
measurement as well as the gain AV. This stability of HEMT characteristics and
performance at 1 K and 4 K are observed for all HEMT geometries.

Even if the power dissipation has to be limited to match the dilution fridge
performance, a reasonable gain must be achieved to mitigate the contraints on
300K electronics. With Vds = 100mV the saturation is well established for all
Vgs. At Ids = 0.15mA we have a low dissipation of 15 $\mu$W, a
transconductance of 10 mS, an output conductance of 0.2 mS and a voltage gain
of 8 (which corresponds to an intrinsic gain gm/gd of $\sim$ 50). This working
point will be used as a benchmark in the following.

The voltage and current noise have been studied in details for all geometries
using the methodology presented in HEMT_YongAPL . A summary on Ricochet
constraints is presented in HEMT_LTD18 . The voltage noise and current noise
magnitudes vary as $1/\sqrt{C_{gs}}$ and $\sqrt{C_{gs}}$ respectively. As a
result the 5 pF device is well adapted to ionization measurement with total
input capacitance of about 20 pF, while the 100 pF and 230 pF devices are well
fitted to Ge-NTD heat sensors up to few tens of M$\Omega$. With such input
impedance the current noise contribution of the 5 pF is negligible with
respect to its voltage noise and the current noise contribution of the 100 pF
and 230 pF devices are negligible with respect to the thermal noise of the Ge-
NTD sensor.

Figure 2: Voltage noise Power Spectral Density at 3 working points for T, Vds
and Ids at 4K, 100 mV, 1 mA; 4K, 100 mV, 0.5 mA and 1K, 100 mV, 0.15 mA
respectively. The noise PSD at 4K are provided by CryoHEMT and are not
corrected from a gain variation below 10 Hz due to the bias output impedance.
The noise PSD at 1K have been measured at Lyon with a gain stable down to 2
Hz. Left: Cgs = 5 pF HEMT device. Right: Cgs = 100 pF and Cgs = 230 pF HEMT
devices. Thick black curve is the targeted noise expected from HEMT_LTD18 .
(Color figure online.)

Voltage noise measurements of 65 Cgs = 5 pF devices and 22 Cgs = 100-230 pF
devices have been provided by the CNRS/C2N laboratory. The noise is measured
at 4.2 K in an He bottle with Vds = 100 mV and Ids = 0.5 and 1 mA. 13 of the
100-230 pF HEMT and 12 of the 5 pF have then been measured at 1K at the low
dissipation working point benchmark in the IP2I dilution fridge for
comparison. Results are show on Fig. 2. The noise has been measured in common
source configuration.

The results for the 5 pF HEMTs are shown on Fig. 2 (Left). The noise level is
the same for the 3 configurations (4K-1mA, 4K-0.5mA, 1K-0.15mA) without any
outliers. The voltage noise is compared to the 5 pF voltage noise model
developed for Ricochet HEMT_LTD18 . For a given input impedance, the detector
input sensitivity is known and the optimal filtering formalism can be used to
extract the expected baseline resolution HEMT_LTD18 . The model gives a RMS
baseline resolution of $\sigma=$20 eVee for a total detector + cabling
capacitance of 20 pF. All baseline resolutions deduced from the 1 K data are
in the 16 to 25 eVee range and 80 $\%$ lie between 18 to 22 eVee. With a
proper preselection of HEMT and additional measurement at 1K, the Ricochet
objectives to have up to 120 5 pF HEMT with voltage noise as expected by the
model HEMT_LTD18  seems reachable by mid-2022.

The noise measurement at the same working point for the 100 pF and 230 pF
HEMTs are shown on Fig. 2 (Right). The spread between the measurements is
larger compared to the 5 pF HEMT, which could be due to the larger gate
surface subject to more impurities or a worse preselection of the HEMT by
CryoHEMT. The extreme outlier in the 1 K dataset is not explained as it passed
the same quality cut as the other measurements. Unlike the ionization signal,
the detector heat sensor impedance is not sufficient to predict the heat
sensitivity. To extract baseline resolutions from the noise PSD we thus
consider a typical pulse shape for the heat signal and normalise the result to
an arbitrary sensitivity. The $\sigma=$ 18 eV RMS resolution obtained in
EDW_Surf  with Si-JFET based electronics have been obtained with a Ge detector
of same mass as planed for Ricochet and a sensitivity of $\sim$ 1.5
$\mu$V/keV. With this sensitivity the heat baseline resolution predicted with
the 1 K data are then in the 16-20 eV range. In fact most of the HEMT tested
have a noise level higher than expected if compared to the model developed in
HEMT_LTD18  at the origin of the 10 eV goal. To meet the 10 eV goal Ricochet
is developing detector with higher sensitivity and works are in progress in
collaboration with C2N and CryoHEMT to have a batch of thirty 230 pF HEMT all
with the expected noise levels.

##  3 HEMT-based charge amplifier for the ionization channel

Proper design of the cold and 300 K parts of the amplification chain is
required to ultimately be limited only by the input HEMT noise. Previous work
BerkeleyNIM  has demonstrated a 91 eVee RMS baseline ionization resolution
with a HEMT-based cryogenic charge amplifier coupled to a 240 g, 150 pF CDMS-
II cryogenic germanium detector, nearly matching the model predictions.
However, the complexity required to implement a HEMT-based fully cryogenic
charge amplifier for the hundred of ionization channels present in Ricochet
requires to explore amplifier topologies where the use of cryogenic HEMTs is
limited to the input transistor. The ionization channel also needs to be
linear between the threshold energy up to tens of MeV expected from high
energy gammas and muons. The gain must be stable at the $\%$ level to allow
for a good reconstruction of the charge signal. For this purpose, the basic
solution is loop topology thanks to a high open loop gain given by a low noise
op amp. The feedback can be on the HEMT gate or on its source. Main issues can
arise from possible oscillations and instabilities induced by the long cabling
between the 1 K cold stage and the 300K electronics. For Ricochet the 300 K
stage will be more than 2 m away from the 1 K stage. However the constraints
on the bandwidth is low, it has to match the detector - muon veto coincidence
window, a limited 40 kHz bandwidth is sufficient according to our simulation
HDRJulien . There are no contraints on the low frequency cut as long as the
resolution goal is met; as the signal/noise ratio is high at low frequency, a
cut off at few Hz is expected.

Figure 3:  Schematics of the charge (Left) and heat (Right) amplifier. Values
of resistance and capacitance are optimized for a 20 pF detector + cabling
input capacitance and a GeNTD heat sensor biased at about 10 M$\Omega$. See
text for details. (Color figure online.)

Fig.3 (Left) shows the schematic of the first version of the Ricochet charge
amplifier. DC biases are made using low noise (6 nV/$\sqrt{Hz}$) DACs (not
shown on the Figure). The source is grounded, Vd and Ids are fixed by the op
amp U1 and the R1, R2, R3 resistor network to 100 mV and 0.15 mA respectively.
The gate is biased thanks to a DC feedback loop of the global amp output
through the high value R9 resistor (10 G$\Omega$) at 1K. The output voltage
offset can be adjusted through R8. The bias noises are attenuated by the
filtering provided by R6-R7-C5 and R8-R7-C5 respectively. The R6/R7=100
divider in the loop provides a boost to R9 which has then an effective value
multiplied by 100 up to the 0.34 Hz low frequency cutoff given by R7 and C5.

The AC response of the amplifier can be described as follows. High open loop
gain is given by the HEMT in series with the low noise op amps U1 and U2
(LT1007 from Linear Technology). The transient current at the drain and
negative input of U1 is gm * $\delta$Vgs. As long as the U1 open loop gain is
high this current is forced to pass through the feedback impedance of C4
connecting the negative input and the output of U1. Op amp U2 is mounted as an
inverter. The open loop gain of the HEMT + U1 + U2 chain is thus:

$G_{OL}=\frac{-g_{m}}{2\pi f\cdot C4}\approx\frac{-10\cdot 10^{6}}{f}$ (1)

The U2 output is then connected to the HEMT input through the AC feedback
provided by the C2-C3-C6 capacitor network, which acts as an effective
feedback capacitance $C_{fb}=C2/(1+C2/C3+C6/C3)=C2/5$ connecting the U2 output
and the HEMT input. The Close Loop gain is :

$G_{cl}=\frac{-\alpha}{1+\alpha\beta}=\frac{-\alpha}{1+\alpha\left(\frac{C_{fb}}{C1+C_{fb}}\right)}\approx\frac{-C1}{C_{fb}}\approx\frac{5\cdot
C1}{C2}\approx-100$ (2)

where $\alpha$ is the absolute value of the open loop gain, $\beta$ is the
feedback factor, and $C1$ is the detector + cabling capacitance. The
approximation holds as long as $\alpha\beta\gg 1$ and $C1\gg C_{fb}$.

The first prototype of this charge amplifier has been tested à 4.2 K in an
helium bottle. R9 was a 1 G$\Omega$ resistor (HVC0805 from Ohmite) with a
measured value of $\sim$ 2 G$\Omega$ at 4.2K. Results are presented on Fig. 4
(Top Panels).

The linearity was tested by injecting a known sinusoidal (1 kHz) voltage at
the input via a 1 pF capacitor. The amplitude was varied up to 85 mV, which
corresponds to about 32 MeVee on a 20 pF input capacitance. Fig. 4 (Top Left)
shows that the linearity is excellent, with a stable gain in all the dynamic
up to the output saturation occurring at 15 V. The feedback loop is working
nicely to stabilize the gain, getting rid especially of the important
transconductance variation at large amplitude. The gain is measured to be 115
for an expected value of 100. This may come from stray capacitance in the test
PCB and variation of the 20 pF dummy detector capacitance from 300 K to 4K
even if COG-NPO material was used as the dielectric.

The bandwidth was measured by varying the frequency of a 1mVpp sine wave.
Three working point have been explored with Vds = 100 mV and Ids = 1 mA, 0.5
mA and 0.15 mA to measure the influence of the transconductance. Data are
compared to the approximated gain expected from Eq.1 and Eq.2 on Fig. 4 (Top
Center). The low frequency cut is well explained by the DC feedback frequency
cut given by R9*100 and C2/5 (4 Hz for 2 G$\Omega$ and 0.2 pF). At frequency
above 10 kHz we measured a gain slightly higher than expected for the 3
working points. This is due to the approximation done in Eq.1 where the gain
variation of U1 with frequency is not included. The measured high frequency
cut vary from 35 kHz to 80 kHz, well within the Ricochet goals.

A stream of few minutes of output data have been recorded at sampling of 200
kHz. The data are first corrected from the gain relative to the amp input and
the noise PSDs are extracted after few basic quality cuts. Fig. 4 (Top Right)
shows that the noise is dominated by the thermal noise of R9 at 4.2 K. With
this high thermal noise the expected baseline resolution is 40 eVee RMS.

Future work will consist of tests at 1 K with a dummy detector and an actual
20 pF detector at 20 mK few cm away from the HEMT input to limit the stray
capacitance. The feedback resistor R9 will be replaced by an active reset
scheme based on low capacitance HEMT, CMOS ASICs or AsGa diode to reach the 20
eVee needed for the Ricochet CryoCube.

Figure 4:  Top Panels: Performance of the charge amplifier at 4.2K with a 2
G$\Omega$ feedback resistance. Bottom Panels: Performance of the voltage heat
amplifier at 4.2K and 1K. NTD dummy resistance shorted (R1 = 0$\Omega$). Left:
1 kHz sine injected at the amp input to measure to linearity of the gain up to
85 mV (eq. to about 32 MeV) for the charge amplifier and 16 mV (eq. to about
15 MeV) for the heat amplifier. Center: Bandwidth measurement at 3 working
points (Vds = 100 mV and Ids = 0.15, 0.5 and 1 mA). Top Right: Noise PSD of
the charge amplifier at 4K 100 mV 0.15 mA. Various contributions to the noise,
as expected by HEMT_LTD18 . Bottom Right: Noise PSD of the heat amplifier at
4K, 100 mV, 0.15 mA and 1K, 100 mV, 0.15 mA are compared to the voltage noise
of the same HEMT provided by CryoHEMT and the 100 pF HEMT noise model from
HEMT_LTD18  (Color figure online.)

##  4 HEMT based voltage amplifier for the heat channel

The heat amplifier shown on Fig. 3 (Right) uses some of the same features of
the charge amplifier. The second stage of amplification at 300 K is based on
the same low noise op amp to provide the needed high open loop gain thanks to
its feedback capacitance C1. The HEMT is biased in the same way. Vd and Ids
are kept constant and fixed by R4, R5, R6 and op amp U1. The NTD is DC coupled
to the HEMT input. This allows a direct bias of the gate through the R14 and
R15 resistors and the NTD itself. The R12-R13-R16 network is added to keep the
NTD bias constant while the gate bias varies. The HEMT source is self-biased
with the current flowing through the low value resistor R3. The HEMT is used
as a differential device, the amplification loop is closed by connecting the
output of U1 to the HEMT source. As the feedback is on the source there is no
need of inverter compared to the charge amplifier. Thanks to this feedback Vs
follows Vg to keep Ids constant. The amplification scheme is equivalent to a
non inverter amplifier with the HEMT gate at the positive input and the source
at the negative input. In the bandwidth of the amplifier where the open loop
gain is high enough the closed loop voltage gain is 1+R9/R3 = 501. Some
subtleties are added to improve linearity and gain stability: R7-R6 adds the
small drain variation on the feedback loop with the same gain, so that Vds is
always kept constant. The cryogenic cabling can have a large impedance and
modify slightly the gain due to the addition of resistance in series with R9.
To avoid this effect U2 is mounted as a Howland source: the current flowing
through R3 is independent on the resistance between R9 and R3.

The first prototype of this heat amplifier have been tested both at 4.2 K in
an Helium bottle and at 1K in a dilution fridge. The R1 dummy NTD resistance
was shorted in order to measure only the voltage noise of the HEMT. As for the
charge amplifier the data taking focus on gain linearity, bandwidth and noise
measurements. The results are presented on Fig. 4 (Lower Pannels).

The linearity was tested by injecting a 1 kHz sine voltage at the HEMT input
via the Ge-NTD bias line. The amplitude was varied up to 16 mV, which
corresponds to about 15 MeV for a typical detector sensitivity. Fig. 4 (bottom
Left) shows that the linearity is excellent, with a stable gain in all the
dynamic up to the output saturation. As for the charge amplifier the feedback
loop is working nicely to stabilize the gain. The gain is measured to be 498
for an expected value of 501.

The bandwidth was measured in a same way as for the charge amplifier at the
same working points. Data are compared to the approximated gain expected from
the open loop and closed loop gain on Fig. 4 (Bottom Center). The amplifier is
DC coupled to the Ge-NTD, thus there is no low frequency roll-off. For the
same reason as for the charge amplifier, at frequency above few kHz we
measured a gain higher than expected for the 3 working points. The measured
high frequency cut vary from 15 kHz to 40 kHz, well within the Ricochet goals.

The voltage noise of the amplifier has been measured with the dummy Ge-NTD
sensor shorted (R1 = 0$\Omega$). Fig. 4 (Bottom Right) shows that the noise
plateau of the 1K measurement in the few 100 Hz- few kHz frequency range is
$\sim$ 0.5nV/$\sqrt{Hz}$. This is bit higher than the HEMT intrinsic voltage
noise but the thermal noise of the Ge-NTD (few M$\Omega$ at 10 mK) will be
higher anyway. At lower frequencies the 1 K data follow the intrinsic HEMT
voltage noise up to few Hz, demonstrating that the voltage noise of the heat
amplifier is adding a negligible noise contribution.

Tests are ongoing with the input resistor at high impedance to measure the
current noise contribution of the amplifier and with a detector prototype to
demonstrate the 10 eV (RMS) heat baseline resolution.

##  5 Conclusion :

The development of HEMT-based amplifiers for charge and heat channels
represents a clear path to achieve the energy resolution, linearity and
bandwidth requirements of Ricochet for pursuing new physics in the CE$\nu$NS
sector. Future work will address the issues of very low capacitance cabling,
active reset schemes and heat sensitivity improvement needed to reach the 20
eVee ionization and 10 eV heat baseline (RMS) resolution goals.

###### Acknowledgements.

This work was supported by the LabEx Lyon Institute of Origins
(ANR-10-LABX-0066) of the Universite de Lyon in the framework ”Investissements
d’Avenir” (ANR-11-IDEX-00007) and has received funding from the European
Research Council (ERC) under the European Union’s Horizon 2020 research and
innovation program under Grant Agreement ERC-StG-CENNS 803079. We thank the
EDELWEISS collaboration for the help and useful discussion in performing this
work.

## References

  * (1) Observation of Coherent Elastic Neutrino-Nucleus Scattering, D. Akimov et al. (COHERENT collaboration), Science 357 (2017) 1123-1126, https://doi.org/10.1126/science.aao0990
  * (2) S. Hertel et al, Ricochet Progress and Status, To be published in J. Low. Temp. Phys. Special Issue for the 19th International Workshop on Low Temperature Detectors 19-29 July 2021 - Virtual event hold by NIST.
  * (3) Exploring CEvNS with NUCLEUS at the Chooz Nuclear Power Plant, G. Angloher et al. (NUCLEUS collaboration), https://arxiv.org/abs/1905.10258
  * (4) Background studies for the MINER Coherent Neutrino Scattering reactor experiment Author links open overlay panel, G.Agnolet et al., Nucl. Instrum. Meth. A853 (2017) 53. https://doi.org/10.1016/j.nima.2017.02.024
  * (5) https://www.ill.eu for general information on ILL.
https://www.ill.eu/reactor-and-safety/high-flux-reactor/technical-
characteristics for reactor information.

  * (6) T. Salagnac et al, Optimization and performance of the CryoCube detector for the future Ricochet low-energy neutrino experiment, J. Low. Temp. Phys. This Special Issue (2021)
  * (7) EXCESS Workshop June 2021 https://indico.cern.ch/event/1013203/ summary paper to be published
  * (8) E. Armengaud, et al., Performance of the EDELWEISS-III experiment for direct dark matter searches, J. Instrum. 12 (08) (2017) P08010. https://doi.org/10.1088/1748-0221/12/08/P08010
  * (9) Searching for low-mass dark matter particles with a massive Ge bolometer operated above ground, E. Armengaud et al. (EDELWEISS Collaboration), Phys. Rev D 99, 082003. https://doi.org/10.1103/PhysRevD.99.082003
  * (10) A. Juillard et al, Low-Noise HEMTs for Coherent Elastic Neutrino Scattering and Low-Mass Dark Matter Cryogenic Semiconductor Detectors, J. Low. Temp. Phys. 199(3):793-806, May 2020. https://doi.org/10.1007/s10909-019-02269-5
  * (11) Q. Dong, et al., Ultra-low noise high electron mobility transistors for high-impedance and low-frequency deep cryogenic readout electronics, Appl. Phys. Lett. 105 (2014) 013504. http://dx.doi.org/10.1063/1.4887368.
  * (12) Y. Jin et al., Ultra-low noise HEMTs for deep cryogenic low-frequency and high-impedance readout electronics, 12th IEEE International Conference on Solid-State and Integrated Circuit Technology (ICSICT) 2014. http://dx.doi.org/10.1109/ICSICT.2014.7021379.
  * (13) http://cryohemt.com/
  * (14) A. Phipps et al., A HEMT-based cryogenic charge amplifier with sub-100 eVee ionization resolution for massive semiconductor dark matter detectors, Nuclear Inst. and Methods in Physics Research, A 940 (2019) 181. https://doi.org/10.1016/j.nima.2019.06.022
  * (15) https://tel.archives-ouvertes.fr/tel-03259707 Searching for Dark Matter and New Physics in the Neutrino sector with Cryogenic detectors, Julien Billard, Habilitation à Diriger des Recherches. Jan 2021


# In BLOOM: Creativity and Affinity in Artificial Lyrics and Art

Evan Crothers,1 Herna Viktor,1 Nathalie Japkowicz2

###### Abstract

We apply a large multilingual language model (BLOOM-176B) in open-ended
generation of Chinese song lyrics, and evaluate the resulting lyrics for
coherence and creativity using human reviewers. We find that current
computational metrics for evaluating large language model outputs (MAUVE) have
limitations in evaluation of creative writing. We note that the human concept
of creativity requires lyrics to be both comprehensible and distinctive — and
that humans assess certain types of machine-generated lyrics to score more
highly than real lyrics by popular artists. Inspired by the inherently
multimodal nature of album releases, we leverage a Chinese-language stable
diffusion model to produce high-quality lyric-guided album art, demonstrating
a creative approach for an artist seeking inspiration for an album or single.
Finally, we introduce the MojimLyrics dataset, a Chinese-language dataset of
popular song lyrics for future research.

## Introduction

Modern music production — which spans elements including musical composition,
songwriting, album artwork, and music videos — is a tremendous domain for
exploring creativity across modalities. This work focuses on better
understanding subjective human assessment of creative text produced by large
language models (LLM), demonstrating that mimicry of an existing sample
dataset may not always be the best approach when considering human
preferences.

Towards this end, we use BLOOM (BigScience 2022), a multilingual LLM with 176B
parameters, to generate Chinese song lyrics with varying sampling parameters.
We focus on top-$p$ sampling, as this method has been found to outperform
top-$k$ and beam search in prior research (Holtzman et al. 2019). In order to
assess the quality of the generated lyrics, we task human reviewers to provide
feedback on lyrical coherence, creativity, and enjoyment. Using this data, we
demonstrate the relationship between creativity and coherence in human-
assessed lyric quality, and highlight differences between subjective human
assessment and computational methods such as Mauve (Pillutla et al. 2021).

Beyond comparison of creativity measures, we also dive into new emerging
multimodal possibilities related to AI-driven creativity. Music today is often
accompanied by cover art that evokes the aesthetic of the piece. This art,
previously displayed on record covers and lyric booklets, remains prominently
displayed on music apps and streaming platforms. While album cover art may
reflect a collection of songs (an album, LP, or EP), digital music publishing
allows for easy release of singles, and cover art is increasingly used to
accompany individual tracks (Leight 2018). Using our lyrics, we use the
Chinese-language Taiyi stable diffusion model (Wang et al. 2022) to generate
an album image inspired by the LLM-generated lyrics. The demonstrated workflow
may be used for inspiration by songwriters seeking a unified lyrical and
visual concept for a single or album, as well as being a source of
entertainment to music fans.

## Related Work

Previous work has used LLMs to generate classical Chinese poetry (Liao et al.
2019). Work has also been performed which uses GPT-2 as a generation system
for various types of creative Chinese writing, contemporary song lyrics among
them (Zhang et al. 2022). Related work has used an image-generation network as
an intermediate step in generating non-lyrical creative text, based on the
idea of an author “visualizing” while they work (Zhu et al. 2022). In English,
research has similarly been done on LLM lyric generation (Rodrigues et al.
2022), including lyrics conditioned on a song’s melody (Chen and Lerch 2020).

Our work is distinct from previous work in that it focuses on zero-shot lyric
generation with a massive multilingual model (BLOOM), is the first to our
knowledge to benchmark lyrics against evaluation metrics that strongly
correlate to human assessment (Mauve), and most importantly, presents an
investigation into the relationship between computational assessment of open-
ended text quality, and subjective measurements. In this department, we are
inspired by previous work on quantifying creativity that found that KL
divergence has a strong correlation with human-assessed creativity for word
pairs (Kuznetsova, Chen, and Choi 2013).

Finally, this work is also the first research work to apply diffusion models
as part of a unified approach for generating coordinated cover art in
association with song lyrics. All models and code and from our work will be
made available open-source to enable future
research111https://github.com/ecrows/in-bloom.

## Methodology

We organize our methodology into three major parts: 1) collecting and cleaning
Chinese lyrics found online to create a novel dataset of popular Chinese
lyrics for use with Mauve, 2) performing lyric generation effectively with the
BLOOM-176B LLM, and 3) utilizing the Taiyi stable diffusion model to generate
high-quality art to accompany the lyrics.

### Dataset

For comparison to LLM-generated lyrics, we collect a dataset of song lyrics
for popular Chinese songs from the website Mojim.com. Mojim is a popular
lyrics sharing website that focuses on Chinese song lyrics. We request the top
popular artists in the “male”, “female”, and “group” categories, fetch all the
albums associated with each artist, and download the lyrics for every song in
each album. Metrics on the dataset can be found in Table 1.

Data downloaded from Mojim.com is in raw HTML format and must be cleaned to
extract raw lyrics. This process is summarized as follows.

  * •

Removal of HTML tags, as well as the removal of markup symbols not typically
part of the lyric text, such as “*” and “#” and various unicode equivalents.

  * •

Removal of site-specific text characteristics, such as: a recurring line in
Chinese which translates to “Find more lyrics at Mojim.com”, dividing lines
composed of hyphens, non-standard labels placed at the start of the song.

  * •

Compress instances of many subsequent newline characters into just two newline
characters, to reduce excess whitespace which may emerge after removing tags,
annotations, and symbol markup.

Songs | Artists | Tokens | Size
---|---|---|---
39747 | 230 | 14,406,854 | 38.4MB
Table 1: Statistics for the MojimLyrics Dataset

### Lyric Generation

We generate zero-shot samples using the multilingual BLOOM-176B model
(BigScience 2022) at varying values of $p$ for top-$p$ nucleus sampling
(Holtzman et al. 2019). By adjusting the probability mass used to sample the
next token, we produce varied lyric sequences that can be used to analyze
quantitative and subjective text characteristics. Our evaluation of lyrics
focuses on two major approaches:

  * •

Computational Quality: We measure computational quality via commonly-used
metrics that target language model degeneration (repeated $n$-grams, diversity
of tokens, distinct $n$-grams), as well as distributional information via
Mauve (Pillutla et al. 2021).

  * •

Subjective: Subjective measures of coherence are collected via a carefully
survey on Amazon Mechanical Turk. These measures include an assessment of how
creative, coherent, and enjoyable the lyrics are.

Previous research on quantifying creativity indicates that for measuring
perceived creativity of word pairs $\\{w_{1},w_{2}\\}$, the KL divergence of
$\\{w_{1},w_{1}w_{2}\\}$ is among the most effective measures (Kuznetsova,
Chen, and Choi 2013).

Mauve is designed to jointly capture two types of errors between the output
distribution $Q$ of a model, and the human distibution $P$. 1) Type I errors,
where $Q$ places high mass on text which is unlikely under $P$; and, 2) Type
II errors, where Q cannot generate text which is plausible under $P$. Mauve
does this by summarizing Type I and Type II errors using Kullback–Leibler (KL)
divergences, which are measured softly using a mixture distribution. Due to
the similarity between Mauve’s computation and previous work on creativity
quantification, it is possible that Mauve may partially reflect subjective
human creativity assessment, which may contribute to Mauve’s strong
performance for evaluating generated text quality more generally.

Additional text quality metrics for assessing language model output are
selected based on previous work (Zhu et al. 2022; Su et al. 2022). These
metrics include rep-$n$ = 1.0 - $\frac{|\text{unique
}n\text{-grams}|}{|\text{total }n\text{-grams}|}$ measuring duplicate
$n$-grams within each generated sequence (Welleck et al. 2020), distinct-$n$ =
$\frac{|\text{unique }n\text{-grams}|}{|\text{length of text}|}$ (Li et al.
2016), and diversity = $\prod_{n\text{=2}}^{4}(1-\text{rep-}n)$ for measuring
diversity of $n$-grams (Su et al. 2022).

Autoregressive language models such as BLOOM perform sampling to select a next
token typically using some variation of top-$k$, beam search, and nucleus
top-$p$ sampling. Due to the strong documented performance of top-$p$ sampling
(Holtzman et al. 2019), we produce BLOOM generations at varying levels of
$p\in[0.80,0.85,0.90,0.95,0.99]$.

Figure 1: BLOOM-176B $p=0.95$, $seed=0$ lyrics mentioning a sword, accompanied
by corresponding stable diffusion generated images selected from the resulting
prompt at $CFG=0.7$, $seed=0$, $batch\\_size=6$

### Album Art Generation

We leverage our generated song lyrics to create accompanying track art
inspired by the song lyrics. As a generative model, we utilize the Chinese-
language Taiyi Stable Diffusion model (Wang et al. 2022; Rombach et al. 2021),
which includes a Chinese-language text encoder. We use this model to produce
1000 samples of album artwork, from seeds 0-999, using the prompt “album art”
in Chinese, followed by our generated lyrics. We perform this at varying
settings of classifier-free diffusion guidance (CFG), where
$CFG\in[4.0,7.0,10.0]$. Images are generated with 20 sampling steps using the
Euler A method (Rombach et al. 2021). For evaluation, we utilize a dataset of
approximately 4,000 album covers at 512x512 resolution (Surma 2019). We
determine the quality of the generated images by calculating Fréchet Inception
Distance (FID) (Heusel et al. 2017; Seitzer 2020) using the pool 3 activations
from an Inception network (Szegedy et al. 2015) between generated album covers
and the real album dataset.

Figure 2: Album covers generated by Taiyi diffusion model in the style of
popular Chinese artists

## Experimental Setup

Inference on BLOOM-176B was performed via the Huggingface Inference API. All
other calculations (stable diffusion, Mauve, FID) were performed on a machine
with an Intel i7-6800K 12-core CPU, 32GB RAM, and a 24GB VRAM RTX 3090 GPU.
Seeds starting at 0 are used throughout our experiments for reproducibility.

### BLOOM

In order to use BLOOM to generate lyrics in a zero-shot setting, we provide
the following prompt:

> Translation: Below are a song’s Chinese lyrics. \n Lyrics: \n

This prompt was chosen based on the observed result of Chinese language song
lyrics. Shorter prompts such as simply the words for “lyrics” in Chinese more
often resulted in non-song lyrics such as interviews with musicians, and more
specific prompts that mentioned specific artists or too closely resembled
formatting from specific lyric websites had a higher incidence of generating
lyric sequences from real songs. Memorization in large language models is
well-documented and is a significant problem undergoing research (Carlini et
al. 2021). The selected prompt appears to minimize this behaviour, though
future analysis on memorization would be valuable.

Source rep-2 $\downarrow$ rep-3 $\downarrow$ rep-4 $\downarrow$ diversity
$\uparrow$ distinct-2 $\uparrow$ Mauve $\uparrow$ Human 0.021 0.013 0.012
0.954 0.870 - $LLM_{p=0.80}$ 0.120 0.090 0.072 0.743 0.650 0.139
$LLM_{p=0.85}$ 0.095 0.073 0.063 0.786 0.689 0.162 $LLM_{p=0.90}$ 0.072 0.055
0.047 0.835 0.722 0.207 $LLM_{p=0.95}$ 0.054 0.044 0.039 0.868 0.760 0.269
$LLM_{p=0.99}$ 0.040 0.034 0.032 0.897 0.783 0.293

Table 2: Quantitative degeneration metrics and Mauve divergence-based quality
results for BLOOM-176B generated text at varying sampling probability mass $p$

### Amazon Mechanical Turk

Using Amazon Mechanical Turk to evaluate open-ended text generation is
challenging and requires care to perform effectively (Karpinska, Akoury, and
Iyyer 2021), however, such platforms remain useful for their ability to
provide real human feedback, A selection of 60 samples were taken from the
BLOOM generated lyric corpora at each value for $top-p$ nucleus sampling, as
well as lyrics from the MojimLyrics dataset. For each of these 60 samples, 3
different reviewers were assigned to score the lyrics for coherence,
creativity, affinity (how much the reviewer likes the lyrics), and recognition
(whether they believe they may have seen the lyrics before). Answers for each
question are converted into normalized scores between 0 and 10.

Instructions were provided in Simplified Chinese. To encourage close reading,
a starting question requests a short summary of the lyrics, and a final
multiple-choice question tests the reviewer’s basic fluency in Chinese by
placing the option “I can read Chinese” among various other negative options.
Participant responses were removed if they failed to provide a summary or
respond correctly to the literacy-testing question.

The projected task duration was 120 seconds, with payment of $0.44 CAD for a
projected hourly wage of $13.20 CAD. Actual mean task duration was confirmed
to be within the projected task duration. The survey questions are provided in
the GitHub repository.

### MAUVE Calculation

For calculating Mauve, we featurize $p$ with a random sample of 3000 real
lyrics from the MojimLyrics dataset, and featurize $q$ with 3000 generated
lyrics from BLOOM-176B at each probability mass from our nucleus sampling
$p\in[0.80,0.85,0.90,0.95,0.99]$. We set a maximum text length of 128 and set
a batch size of 32 based on the GPU memory available. The seed for the Mauve
calculation is set to 0.

## Results

Results for language model evaluation with Mauve scores and degeneration
metrics can be found in Table 2, while results of stable diffusion CFG
variation on FID can be found in Table 3. The complete dataset of generated
BLOOM lyrics are provided in the GitHub repository. Human assessment of lyrics
can be found in Figure 3.

A sample of generated lyrics and the resulting stable diffusion art can be
found in Figure 1. A wider range of stable diffusion images using the album
art prompt can be found in Figure 2, demonstrating the variety of generations
the model can produce. FID scores calculated between outputs of the diffusion
model and real album art can be found in Table 3.

Figure 3: Mean human scores for subjective attributes of generated text at varying sampling probability mass $p$. Included to the right are reviewer scores for real lyrics. |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---|---
FID$\downarrow$ | 90.21 | 93.84 | 95.91
Table 3: FID with real album dataset of Taiyi stable diffusion generated album
art at varying CFG

On the 10 point normalized scale, mean standard deviation among multiple
responses for the same input sample were observed to be 1.29, 1.68, 1.95, and
2.67 for coherence, affinity, creativity, and recognition respectively.
However, despite the relatively low variance in response scores, measures of
inter-annotator agreement (IAA) are consistently low (i.e., negative
Krippendorff ordinal alpha). Survey responses were highly skewed towards
positive answers. When a reviewer gave a particular sample a low score, often
it was only one reviewer did so. This implies that while reviewers largely
agreed on ratings, there was not a consistent criterion across reviewers for
giving low scores. Obtaining high IAA scores on concepts such as creativity
and personal preference will likely require a larger study with a greater
number of trained reviewers.

## Discussion

Sampling BLOOM for lyrics from a greater portion of the probability mass tends
to improve quantitative metrics of generated lyric quality, as shown in Table
2. As expected, token diversity, distinct-$n$, and rep-$n$ all improve at
increasing $p$ portion of the probability mass. That Mauve score also
continues improving implies a low incidence of Type I errors where the model
output distribution $Q$ places high probability mass on text which is unlikely
under $P$. Notably, even at very high $p$-values, BLOOM-176B generated lyrics
are still more repetitive and less diverse than human lyrics.

Human assessment of lyrics at varying $p$ is charted in Figure 3. Coherence is
highest at low-$p$, where common words are frequently sampled, but with a low
accompanying creativity score. Subjective quality measurements of creativity
and affinity peak around $p=[0.9,0.95]$, and trend sharply downwards at
$p=0.99$ as the model selects less common words and becomes less coherent.
This implies that for a human to recognize work as “creative”, it must be
coherent enough to be understood, but distinctive enough to be interesting.
While Mauve rises alongside subjective creativity from $p=0.80$ to $p=0.95$,
this relationship breaks down at $p=0.99$, implying that Mauve alone is not a
complete replacement for subjective human evaluation.

Finally, it is notable that real popular lyrics uploaded to Mojim.com are at
points deemed less creative, coherent, and likable than BLOOM-176B
generations. Resemblance to a human dataset does not inherently bestow
positive perceived qualities to generated art. While the best-scoring CFG
setting in Table 3 appears to be at CFG=4.0, this measurement alone is not
indicative of subjective human quality.

## Conclusion

In conclusion, we have demonstrated that current computational techniques for
evaluating large language model output (both quantitative degeneration metrics
and Mauve), are insufficient as proxies for subjective assessment of creative
writing. Based on human reviewers, we find that large language models such as
BLOOM-176B can be used to produce zero-shot song lyrics that are evaluated
favourably in comparison to real song lyrics. Inpsired by AI-driven
creativity, we provide a combined approach for generating album art based on
lyrics using diffusion. Finally, we provide a novel dataset of in-the-wild
popular Chinese song lyrics, and the code for repeating our experiments.

### Future Work

We identify the following open problems for future research.

  * •

Quantification or representation of abstract concepts such as creativity, and
shaping generative models to produce outputs that manifest such attributes,
across modalities.

  * •

Analysis of sampling parameters and prompts on characteristics of LLM text
generation, incorporating both Mauve and subjective human assessment
(including a more comprehensive subjective study).

  * •

Deeper study of large multilingual language models such as BLOOM-176B, on
topics such as training data memorization, overparameterization, and fairness.

## Acknowledgments

Thank you to Annie Feng for survey proofreading, to Bowen Gu for past
discussions on lyric generation, and workers at Amazon Mechanical Turk for
participating in this study.

## References

  * BigScience (2022) BigScience. 2022.  BLOOM: A 176B-Parameter Open-Access Multilingual Language Model.
  * Carlini et al. (2021) Carlini, N.; Tramer, F.; Wallace, E.; Jagielski, M.; Herbert-Voss, A.; Lee, K.; Roberts, A.; Brown, T.; Song, D.; Erlingsson, U.; et al. 2021.  Extracting training data from large language models.  In _30th USENIX Security Symposium (USENIX Security 21)_ , 2633–2650.
  * Chen and Lerch (2020) Chen, Y.; and Lerch, A. 2020.  Melody-conditioned lyrics generation with seqgans.  In _2020 IEEE International Symposium on Multimedia (ISM)_ , 189–196. IEEE.
  * Heusel et al. (2017) Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; Klambauer, G.; and Hochreiter, S. 2017.  GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium.  _CoRR_ , abs/1706.08500.
  * Holtzman et al. (2019) Holtzman, A.; Buys, J.; Du, L.; Forbes, M.; and Choi, Y. 2019.  The curious case of neural text degeneration.  _arXiv preprint arXiv:1904.09751_.
  * Karpinska, Akoury, and Iyyer (2021) Karpinska, M.; Akoury, N.; and Iyyer, M. 2021.  The perils of using mechanical turk to evaluate open-ended text generation.  _arXiv preprint arXiv:2109.06835_.
  * Kuznetsova, Chen, and Choi (2013) Kuznetsova, P.; Chen, J.; and Choi, Y. 2013.  Understanding and quantifying creativity in lexical composition.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , 1246–1258.
  * Leight (2018) Leight, E. 2018.  Why your favorite artist is releasing more singles than ever.
  * Li et al. (2016) Li, J.; Galley, M.; Brockett, C.; Gao, J.; and Dolan, W. B. 2016.  A Diversity-Promoting Objective Function for Neural Conversation Models.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 110–119.
  * Liao et al. (2019) Liao, Y.; Wang, Y.; Liu, Q.; and Jiang, X. 2019.  GPT-based Generation for Classical Chinese Poetry.  _ArXiv_ , abs/1907.00151.
  * Pillutla et al. (2021) Pillutla, K.; Swayamdipta, S.; Zellers, R.; Thickstun, J.; Welleck, S.; Choi, Y.; and Harchaoui, Z. 2021.  Mauve: Measuring the gap between neural text and human text using divergence frontiers.  _Advances in Neural Information Processing Systems_ , 34: 4816–4828.
  * Rodrigues et al. (2022) Rodrigues, M. A.; Oliveira, A.; Moreira, A.; and Possi, M. 2022.  Lyrics Generation supported by Pre-trained Models.  In _The International FLAIRS Conference Proceedings_ , volume 35.
  * Rombach et al. (2021) Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2021.  High-Resolution Image Synthesis with Latent Diffusion Models.  arXiv:2112.10752.
  * Seitzer (2020) Seitzer, M. 2020.  pytorch-fid: FID Score for PyTorch.  https://github.com/mseitzer/pytorch-fid.  Version 0.2.1.
  * Su et al. (2022) Su, Y.; Lan, T.; Wang, Y.; Yogatama, D.; Kong, L.; and Collier, N. 2022.  A Contrastive Framework for Neural Text Generation.  _arXiv preprint arXiv:2202.06417_.
  * Surma (2019) Surma, G. 2019.  Album covers images dataset.
  * Szegedy et al. (2015) Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; and Rabinovich, A. 2015.  Going deeper with convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1–9.
  * Wang et al. (2022) Wang, J.; Zhang, Y.; Zhang, L.; Yang, P.; Gao, X.; Wu, Z.; Dong, X.; He, J.; Zhuo, J.; Yang, Q.; Huang, Y.; Li, X.; Wu, Y.; Lu, J.; Zhu, X.; Chen, W.; Han, T.; Pan, K.; Wang, R.; Wang, H.; Wu, X.; Zeng, Z.; Chen, C.; Gan, R.; and Zhang, J. 2022.  Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence.  _CoRR_ , abs/2209.02970.
  * Welleck et al. (2020) Welleck, S.; Kulikov, I.; Roller, S.; Dinan, E.; Cho, K.; and Weston, J. 2020.  Neural Text Generation With Unlikelihood Training.  In _International Conference on Learning Representations_.
  * Zhang et al. (2022) Zhang, L.; Zhang, R.; Mao, X.-X.; and Chang, Y. 2022.  QiuNiu: A Chinese Lyrics Generation System with Passage-Level Input.  In _ACL_.
  * Zhu et al. (2022) Zhu, W.; Yan, A.; Lu, Y.; Xu, W.; Wang, X. E.; Eckstein, M.; and Wang, W. Y. 2022\.  Visualize Before You Write: Imagination-Guided Open-Ended Text Generation.  _arXiv preprint arXiv:2210.03765_.