# Bridging the Gap between Human Motion and Action Semantics via Kinematic
Phrases

Xinpeng Liu1, Yong-Lu Li1, Ailing Zeng2, Zizheng Zhou 1, Yang You1, Cewu
Lu1††footnotemark:
1Shanghai Jiao Tong University, 2International Digital Economy Academy (IDEA)
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Yong-Lu Li and Cewu Lu are the corresponding authors.

###### Abstract

The goal of motion understanding is to establish a reliable mapping between
motion and action semantics, while it is a challenging many-to-many problem.
An abstract action semantic (i.e., walk forwards) could be conveyed by
perceptually diverse motions (walk with arms up or swinging), while a motion
could carry different semantics w.r.t. its context and intention. This makes
an elegant mapping between them difficult. Previous attempts adopted direct-
mapping paradigms with limited reliability. Also, current automatic metrics
fail to provide reliable assessments of the consistency between motions and
action semantics. We identify the source of these problems as the significant
gap between the two modalities. To alleviate this gap, we propose Kinematic
Phrases (KP) that take the objective kinematic facts of human motion with
proper abstraction, interpretability, and generality characteristics. Based on
KP as a mediator, we can unify a motion knowledge base and build a motion
understanding system. Meanwhile, KP can be automatically converted from
motions and to text descriptions with no subjective bias, inspiring Kinematic
Prompt Generation (KPG) as a novel automatic motion generation benchmark. In
extensive experiments, our approach shows superiority over other methods. Our
code and data would be made publicly available here.

##  1 Introduction

Human motion understanding has a wide range of applications, including
autonomous driving (Paden et al., 2016), robotics (Koppula & Saxena, 2013),
and automatic animation (Van Welbergen et al., 2010), making it increasingly
attractive. The core of human motion understanding is to establish a mapping
between the motion space and the action semantics space. The motion space
indicates a space of sequential 3D human representations, e.g., 3D pose or
SMPL (Loper et al., 2015)/SMPL-X (Pavlakos et al., 2019) parameter sequence,
while the action semantic space can be represented as action categories or
sentences described by natural language.

Recently, a growing focus has been on generative mapping from semantics to
motion, including action category-based generation (Petrovich et al., 2021)
and text-based generation (Petrovich et al., 2022; Guo et al., 2022a; Lucas et
al., 2022; Zhang et al., 2022; Tevet et al., 2022b; Chen et al., 2023; Zhang
et al., 2023a). Most of them typically build a mapping that links motion and
semantics either directly or via motion latent, with understated concerns for
intermediate motion-semantic structures. However, these models suffer from
inferior reliability. They cannot guarantee they generated correct samples
without human filtering. Additionally, the existing evaluation of motion
generation is problematic. Widely adopted FID and R-Precision rely on the
latent space from a black-box pre-trained model, which might fail to out-of-
distribution (OOD) and over-fitting cases. There is a long-standing need for
an evaluation method that can cheaply and reliably assess whether a generated
motion is consistent with particular action semantics. We identify the essence
of these as the significant gap between raw human motion and action semantics,
which makes direct mapping hard to learn.

Figure 1:  The huge gap between motion and action semantics results in the
many-to-many problem. We propose Kinematic Phrases (KP) as an intermediate to
bridge the gap. KPs objectively capture human kinematic cues. It properly
abstracts diverse motions with interpretability. As shown, the Phrases in the
yellow box could capture key patterns of walk for diverse motions.

As in Fig. 1, an action semantics can correspond to diverse motions. For
instance, a person could walk in countless ways with diverse motions, either
with arms up or swinging, while action semantics tend to abstract these away
from a walking motion. Additionally, they are robust against small
perturbations, while motion is more specific and complex, with representations
changing vastly when perturbed or mis-captured. Moreover, a motion sequence
could have diverse semantics w.r.t. contexts. Modeling this many-to-many
mapping between motion and semantics is challenging.

To bridge this gap between motion and action semantics, we propose Kinematic
Phrases (KP), an interpretable intermediate representation. KP focuses on the
objective kinematic facts, which are usually omitted by general action
semantics, like left-hand moving forwards then backward. KP is designed as
qualitative categorical representations of these facts. For objectivity and
actuality, KP captures sign changes with minimal pre-defined standards.
Inspired by previous studies on kinematic human motion representation (von
Laban & Lange, 1975; Bartlett, 1997), KP is proposed as six types shown in
Fig. 1, covering joint positions, joint pair positions and distances, limb
angles and directions, and global velocity. Note that, although KP can be
described by natural language, a major difference is that KP is strictly
dedicated to objective kinematic facts instead of coarse actions such as
surrender or fine-grained actions like raise both hands.

We highlight three advantages of KP. First, KP offers proper abstraction,
which disentangles motion perturbations and semantics changes, easing the
learning process. Even though the motion differs significantly, KP manages to
capture walk patterns easily. Second, KP is interpretable, as it can be viewed
as instructions on executing the action, making it easily understandable to
humans. Finally, KP is general, as it can be automatically extracted from
different modalities of human motion, including skeleton and SMPL parameters.
The conversion from KP to text is also effortless.

With KP as an intermediate representation, we first construct a unified large-
scale motion knowledge base. Then, to fully exploit KP and the knowledge base,
we build a motion understanding system with KP mediation. In detail, we learn
a motion-KP joint latent space in a self-supervised manner and then adopt it
for multiple motion understanding applications, including motion
interpolation, modification, and generation. Moreover, leveraging the
interpretability of KP, we propose a benchmark called Kinematic Prompts
Generation (KPG), which generates motion from text prompts converted from KPs.
Thanks to the consistency and convenience of the KP-to-text conversion, KPG
enables reliable and efficient motion generation evaluation.

Our contributions are: (1) We propose KP as an intermediate representation to
bridge the gap between motion and action semantics. (2) We build a novel
motion understanding system using KP and the aggregated large-scale knowledge
base. (3) We propose KPG as a benchmark for reliable and efficient motion
generation evaluation. Promising results are achieved on motion interpolation
and generation tasks. Moreover, extensive user studies are conducted,
verifying the efficacy of our methods, also the consistency between KPG
evaluation and human perception.

##  2 Related Works

Motion Representation. An intuitive motion representation is a sequence of
static pose representations, like joint locations and limb rotations. Efforts
are paid to address the discontinuity of rotation for deep-learning methods
(Zhou et al., 2019; Brégier, 2021). Recent works on parametric body models
(Loper et al., 2015; Pavlakos et al., 2019) enable a more realistic body
representation. Meanwhile, Pons-Moll et al. (2014) proposed Posebits,
representing pose with boolean geometric part relationships. Delmas et al.
(2022; 2023) translates Posebits into text descriptions. These abstract
representations are flexible and insensitive to little perturbations, but
their static nature ignores motion dynamics. Tang et al. (2022) acquire
similar fine-grained descriptions from human annotation, while Xiang et al.
(2022); Athanasiou et al. (2023) adopted large-scale language models. However,
few recognize their potential in bridging the low-level motion and the high-
level action semantics. Phase functions (Holden et al., 2020), Labanotations
(von Laban & Lange, 1975), and learned Motion Words (Aristidou et al., 2018)
were also explored, though limited to specific actions like locomotion and
dancing.

Motion Generation can be conditioned by its prefix/suffix (Hernandez et al.,
2019; Athanasiou et al., 2022; Guo et al., 2023), action categories (Petrovich
et al., 2021; Guo et al., 2020; Xu et al., 2023), or audio (Li et al., 2021a;
b). Text-based motion generation has developed rapidly with the proposal of
text-motion datasets Punnakkal et al. (2021); Guo et al. (2022a). Petrovich et
al. (2022); Guo et al. (2022a); Qian et al. (2023) used VAEs, while Tevet et
al. (2022a); Hong et al. (2022); Lin et al. (2023b) extended the CLIP (Radford
et al., 2021) space to motion. Recently, attention has been paid to diffusion
models (Zhang et al., 2022; Tevet et al., 2022b; Dabral et al., 2023; Wang et
al., 2023). Azadi et al. (2023) adopted a U-Net structure. Zhang et al.
(2023b); Petrovich et al. (2023) explored retrieval-based methods.
Karunratanakul et al. (2023) aimed at controllable generation, while Yuan et
al. (2023) introduced physical constraints. However, most approaches still
suffer from the gap between motion and action semantics. Lucas et al. (2022);
Guo et al. (2022b); Zhang et al. (2023a); Chen et al. (2023); Zhou & Wang
(2023); Zhong et al. (2023); Kong et al. (2023) adopted (VQ-)VAE-compressed
motion representation as mediation, while in the current data-limited
situation, we identify that this single-modality compression might be sub-
optimal. Instead, KP could alleviate this by introducing explicit semantic-
geometric correlation.

##  3 Kinematic Phrase Base

###  3.1 Kinematic Phrases

Kinematic Phrases abstract motion into objective kinematic facts like left-
hand moves up qualitatively. We take inspiration from previous kinematic
motion representations (von Laban & Lange, 1975) and qualitative static pose
representations (Delmas et al., 2022; Pons-Moll et al., 2014), proposing six
types of KP to comprehensively represent motion from different kinematic
hierarchies: For joint movements, there are 36 Position Phrases (PPs). For
joint pair movements, there are 242 Pairwise Relative Position Phrases (PRPPs)
and 81 Pairwise Distance Phrases (PDPs). For limb movements, there are 8 Limb
Angle Phrases (LAPs) and 33 Limb Orientation Phrases (LOPs). For whole-body
movements, there are 3 Global Velocity Phrases (GVPs). KP extraction is based
on a skeleton sequence $X=\\{x_{i}|x_{i}\in\mathcal{R}^{n_{k}\times
3}\\}_{i=1}^{t}$, where $n_{k}$ is the number of joints ($n_{k}=17$ here),
$x_{i}$ is the joint coordinates at $i$-th frame, and $t$ is the sequence
length. Note that $x_{i}^{0}$ indicates the pelvis/root joint. For each
Phrase, a scalar indicator sequence is calculated from the skeleton sequence.
Phrases are extracted as per-frame categorical representations w.r.t.
indicator signs. Unlike previous efforts (Pons-Moll et al., 2014; Delmas et
al., 2022), we limit the criteria of KP as the indicator signs to minimize the
need for human-defined standards (e.g., numerical criteria on the closeness of
two joints) for objectivity and actuality. Fig. 2 illustrated the extraction
procedure.

Figure 2: Six types of KP from four kinematic hierarchies are extracted from a
motion sequence. A scalar indicator $s_{i}$ is calculated per Phrase per
frame. Its sign categorizes the corresponding Phrase.

Reference Vectors are first constructed, indicating right, upward, and forward
directions from a human cognitive view. We aim at the egocentric reference
frames that human tends to use when performing actions. The negative direction
of gravity is adopted as upward vector $r^{u}$, the vector from left hip to
right hip is adopted as right vector $r^{r}$, and the forward vector is
calculated as $r^{f}=r^{r}\times r^{r}$. These vectors of each frame are
denoted as $R^{\cdot}=\\{r^{\cdot}_{i}\\}_{i=1}^{t}$.

Position Phrase (PP) focuses on the movement direction of joint $x^{j}$ w.r.t.
reference vector $R^{\cdot}$. The indicator for PP at $i$-th frame is
calculated as

$s_{i}^{(j,\cdot)}=\langle(x_{i}^{j}-x_{i}^{0}),r^{\cdot}_{i}\rangle-\langle(x_{i-1}^{j}-x_{i-1}^{0}),r^{\cdot}_{i-1}\rangle.$
(1)

The sign of $s_{i}^{(j,\cdot)}$ categorizes PP into moving along/against
$R^{\cdot}$, or relatively static along $R^{\cdot}$ for indicators with small
amplitudes. After filtering, 36 different PPs are extracted.

Pairwise Relative Position Phrase (PRPP) describes the relative position
between a pair of joints $(x^{j},x^{k})$ w.r.t. reference vector $R^{\cdot}$.
PRPP indicator at $i$-th frame is
$s_{i}^{(j,k,\cdot)}=\langle(x_{i}^{j}-x_{i}^{k}),r^{\cdot}_{i}\rangle$. For
(L-Hand, R-Hand) and forward vector $R^{f}$, PRPP could be L-Hand behind/in
front of R-Hand according to the sign of $s_{i}^{(j,k,\cdot)}$. After
filtering, 242 PRPPs are extracted.

Pairwise Distance Phrase (PDP) describes how the L2 distance between a pair of
joints $(x^{j},x^{k})$ changes. The indicator for PDP is calculated as

$s_{i}^{(j,k)}=\|x_{i}^{j}-x_{i}^{k}\|_{2}-\|x_{i-1}^{j}-x_{i-1}^{k}\|_{2}.$
(2)

The sign of $s_{i}^{(j,k)}$ categorizes PDP into moving closer/away, or
relatively static. After dropping joint pairs in the skeleton topology, such
as the hand and elbow, 81 PDPs are extracted.

Limb Angle Phrase (LAP) targets at the change of bend angle between two
connected limbs $(x^{j},x^{k})$ and $(x^{j},x^{l})$. The indicator for LAP is
calculated as

$s_{i}^{(j,k,l)}=arccos(\langle
x^{k}_{i}-x^{j}_{i},x^{l}_{i}-x^{j}_{i}\rangle)-arccos(\langle
x^{k}_{i-1}-x^{j}_{i-1},x^{l}_{i-1}-x^{j}_{i-1}\rangle).$ (3)

LAP describes the limb chain $(x^{j},x^{k})$-$(x^{j},x^{l})$ as bending or
unbending. 8 LAPs are extracted.

Limb Orientation Phrase (LOP) describes the orientation of the limb
$(x^{j},x^{k})$ w.r.t. $R^{\cdot}$, note that $x^{k}$ is the distal limb. The
scalar indicator for LOP is calculated as $s_{i}^{(j,k,\cdot)}=\langle
x^{k}_{i}-x^{j}_{i},r^{\cdot}_{i}\rangle$. The sign of $s_{i}^{(j,k,\cdot)}$
categorizes the LOP into limb $(x^{j},x^{k})$ pointing along/against
$R^{\cdot}$, or a placeholder category for those with little magnitude. 33
LOPs are extracted.

Global Velocity Phrase (GVP) describes the direction of global velocity with
respect to $R^{\cdot}$. The indicator is calculated as $s_{i}^{\cdot}=\langle
x^{0}_{i+1}-x^{0}_{i},r^{\cdot}_{i}\rangle$. The three categories are moving
along/against $R^{\cdot}$, or static along $R^{\cdot}$ according to the sign
of $s_{i}^{\cdot}$.

These result in 403 Phrases in total, covering motion diversity and
distribution from various levels. While we clarify that these Phrases do not
rule out the possibility of other possible useful potentials.

Dataset | Mot. Rep. | #Seqs | #Actions | Text
---|---|---|---|---
AMASS (Mahmood et al., 2019) | SMPL-X | 26k | 260 | ✓
GRAB (Taheri et al., 2020) | SMPL-X | 1k | 4 | ✓
SAMP (Hassan et al., 2021) | SMPL-X | 0.2k | N/A | ✓*
Fit3D (Fieraru et al., 2021) | SMPL-X | 0.4k | 29 | ✓
CHI3D (Fieraru et al., 2020) | SMPL-X | 0.4k | 8 | ✓
UESTC (Ji et al., 2018) | SMPL | 26k | 40 | ✓
AIST++ (Li et al., 2021a) | SMPL | 1k | N/A | ✓*
BEHAVE (Bhatnagar et al., 2022) | SMPL | 0.3k | N/A | ✓*
HuMMan (Cai et al., 2022) | SMPL | 0.3k | 339 | ✓
GTAHuman (Cai et al., 2021) | SMPL | 20k | N/A | x
Motion-X(Lin et al., 2023a) | SMPL-X | 65k | N/A | ✓
Sum | - | 140k | 680+ | -

Table 1: Statistics of Kinematic Phrase Base. Mot. Rep. indicates motion
representation. “✓*” means texts are generated from the attached additional
information instead of human annotation.

###  3.2 Constructing Kinematic Phrase Base

KP enables us to unify motion data with different formats to construct a
large-scale knowledge base containing motion, text, and KP. Motion sequences
of different representations are collected, including 3D skeleton sequences
and SMPL (Loper et al., 2015)/SMPL-X (Pavlakos et al., 2019) parameter
sequences. The sequences are first re-sampled to 30Hz and rotated so that the
negative direction of the z-axis is the gravity direction. Then, the sequences
are converted into 3D skeleton sequences for KP extraction as in Sec. 3.1.
Text annotations attached to the sequences are directly saved. For sequences
with action category annotation, the category name is saved. For those with
neither text nor action category, the text information is set from its
attached additional information, like objects for SAMP (Hassan et al., 2021).
Finally, we collect 87k motion sequences from 11 datasets. Detailed statistics
are shown in Tab. 1. More details are included in the appendix.

##  4 Motion Understanding via KP

By motion understanding, we mean both low-level understanding like
interpolation and modification, and high-level understanding like generative
mapping from text to motion. To achieve this, we first learn a motion-KP joint
space with less ambiguity and more interpretability. Then, with this space, we
introduce its application to both low-level and high-level motion-semantics
understanding.

###  4.1 Preliminaries

We first introduce the representation for motion and KP. Motion is represented
as a human pose sequence with $n$ frames as $M=\\{m_{i}\\}_{i=1}^{n}$. In
detail, SMPL (Loper et al., 2015) pose parameters are transformed from axis-
angle format to the 6D continuous representation (Zhou et al., 2019), then
concatenated with the velocity of the root joint, resulting in a
147-dimensional representation per frame. KP is represented by signs of the
indicators.

###  4.2 Joint Space Learning

Model Structure. An overview of our model is illustrated in Fig. 3. Motion VAE
is a transformer-based VAE adapted from Petrovich et al. (2021). The encoder
$\mathcal{E}_{m}$ takes motion $M$ and two distribution tokens
$m_{\mu},m_{\sigma}$ as input, and the outputs corresponding to the
distribution tokens are taken as the $\mu_{m}$ and $\sigma_{m}$ of the
Gaussian distribution. Then, the transformer decoder $\mathcal{D}_{m}$ takes
$z_{m}\sim\mathcal{G}(\mu_{m},\sigma_{m})$ as $K,V$, and a sinusoidal
positional encoding of the expected duration as $Q$. The output is fed into a
linear layer to obtain the reconstructed motion sequence $\hat{M}$. KP VAE
with encoder $\mathcal{E}_{p}$ and decoder $\mathcal{D}_{p}$ resembles Motion
VAE. The sign of $\mathcal{D}_{p}$ output is adopted as the predicted KP
$\hat{C}$.

Figure 3:  We train motion-KP joint latent space in a self-supervised training
manner. KP is randomly masked during training. Reconstruction and alignment
losses are adopted. The joint space could be applied for multiple tasks,
including motion interpolation, modification, and generation.

Notice that the decoders $\mathcal{D}_{m},\mathcal{D}_{p}$ could take
arbitrary combinations of $z_{m},z_{p}$ as input, outputting
$\hat{M}_{\cdot},\hat{C}_{\cdot}$.

Self-supervised Training. With the VAEs, we propose a self-supervised training
strategy to learn motion-KP joint space. As a coherent representation, the
overall representation should not change drastically with a small portion of
KP unknown. Even more, the missing Phrases should be recovered from existing
Phrases. In this view, we randomly corrupt samples during training by setting
a small portion of KP as 0. The training is thus executed in a self-supervised
manner. This helps mine the correlation among different Phrases while also
effectively increasing the robustness of the joint space. Similar to TEMOS
(Petrovich et al., 2022), four losses are adopted: reconstruction loss, KL
divergence loss, distribution alignment loss, and embedding alignment loss.

###  4.3 KP-mediated Motion Understanding

With the joint space, we can perform both low-level and high-level motion
understanding with KP mediation. We introduce three applications to show the
capability of KP, as shown in Fig. 3.

KP-mediated Motion Interpolation Given a corrupted motion sequence
$\tilde{M}$, we extract its corresponding KP sequence $\tilde{C}$, then feed
them to encoders $\mathcal{E}_{m},\mathcal{E}_{p}$ and decoder
$\mathcal{D}_{p}$, resulting in the estimated KP sequence $\hat{C}$. $\hat{C}$
and $\tilde{M}$ are fed into $\mathcal{E}_{m},\mathcal{E}_{p}$ and
$\mathcal{D}_{m}$, resulting in interpolated $\hat{M}$.

Motion Modification Motion modification functions similarly. Motion $M$ is
first extracted into KP sequence $C$. Modifications could be made on $C$
resulting in $\tilde{C}$. Modified motion frames are then masked, getting
$\tilde{M}$. $\tilde{M},\tilde{C}$ are fed into
$\mathcal{E}_{m},\mathcal{E}_{p}$ and $\mathcal{D}_{m}$, getting the
interpolated $\hat{M}$.

KP-mediated Motion Generation. Given text $t$, to generate a motion sequence
from it, we first encode it into latent $z_{t}$ with CLIP text encoder
$\mathcal{E}_{t}$. Direct mapping could be achieved by training the motion
decoder $\mathcal{D}_{m}$ for $\hat{M}=\mathcal{D}_{m}(z_{t})$. We show that
the direct mapping could be impressively improved with our joint space in Sec.
6.4. With KP, we could perform a novel KP-mediated motion generation. We adopt
a vanilla latent diffusion paradigm for KP-mediated text-to-motion tasks. An
extra denoiser is trained to denoise a random noise $z_{p}^{T}$ to KP latent
$z_{p}=z_{p}^{0}$ with $T$ diffusion steps. We then decode KP sequence
$\hat{C}$ from $z_{p}$ with $\mathcal{D}_{p}$. Then, $\hat{C}$ is encoded by
$\mathcal{E}_{p}$, getting distribution $\mathcal{G}(\mu_{p},\sigma_{p})$.
$z_{p}$ is sampled and sent to $\mathcal{D}_{m}$ to generate a motion
sequence. Experiments show that KP could be a promising stepping stone to
mitigate the huge gap from action semantics to motion.

##  5 Kinematic Prompt Generation

With the interpretability and objectivity of KP, we propose a new motion
generation benchmark.

Before that, we first analyze current benchmarks. A crucial aspect of motion
generation evaluation is motion-semantic consistency. The gold standard is
user study. However, it is expensive and inefficient to scale. Early metrics
like MPJPE (Mean Per Joint Position Error) and MAE (Mean Angle Error)
mechanically calculate the error between the generated and GT samples. These
metrics fail to reveal the real ability of generative models: What if the
models memorize GT samples? Or what if the samples are diverse from GT but
also true? FID (Frechet Inception Distance) is adopted to mitigate this issue.
However, it provides a macro view of the quality of all generated samples
without guarantees for individual samples. Guo et al. (2022a) proposed
R-Precision, using a pre-trained text-motion matching model to examine whether
the generated samples carry true semantics. They both rely on the latent space
from a black-box pre-trained model, which is not credible. Besides, models
might learn short paths to over-fit the pre-trained model. Moreover, since
automatic mapping from motion to semantics across their huge gap is still an
unsettled problem, adopting it to evaluate motion generation is not a decent
choice. Moreover, most current motion generation evaluations are performed on
datasets (Guo et al., 2022a; Plappert et al., 2016; Ji et al., 2018) with
considerable complex everyday actions, further increasing the difficulty.

To this end, we propose a novel benchmark: Kinematic Prompts Generation (KPG).
Instead of previous benchmarks focusing on everyday activities or sports, we
take a step back in the complexity of the target action semantics. Based on
KP, KPG focuses on evaluating whether the models could generate motion
sequences consistent with specific kinematic facts given text prompts.

KP | Text prompt samples
---|---
PP |  Left hand moves forwards.
PRPP |  Left hand is below head then above head.
PDP |  Left hand moves away from head.
LAP |  Left arm bends.
LOP |  Left forearm points forwards then backward.
GVP | The person moves forwards.

Table 2:  Text prompts converted from KP. Joint/limb names, prepositions,
verbs, and adverbials could be replaced w.r.t. specific Phrases.

In detail, we convert KP into text prompts with templates as in Tab. 2,
resulting in 840 text prompts. Given prompt $T_{i}\in T$ from Phrase $c_{i}$,
the model generates motion $\hat{M}_{i}$, along with extracted KP
$\hat{C}_{i}$. We calculate Accuracy as $Acc=\frac{1}{|T|}\sum_{T_{i}\in
T}\textrm{1}[c_{i}\in\hat{C}_{i}]$, where $\textrm{1}[\cdot]=1$ if the
expression in $[\cdot]$ is True, otherwise 0. Note that, for
$c_{i}\in\hat{C}_{i}$, $c_{i}$ should keep for more than 5 consecutive frames
to avoid trivial perturbations. Accuracy examines whether the Phrase
corresponding to the given prompt appears in the KP sequence converted from
generated motion. The calculation involves no black-box model thanks to KP,
presenting a fully reliable evaluation pipeline. Also, with the effortless
motion-to-KP conversion, the computation could be conducted automatically.
More details are in the appendix.

##  6 Experiment

Implementation Details. HumanML3D (Guo et al., 2022a) test split is held out
for evaluation, with the rest of KPB for training. During training, the motion
sequences are canonicalized by eliminating the rotation along the z-axis in
the first frame, and the same counter-rotation is applied to the following
frames. Sequences are sampled to 15 FPS and randomly clipped into short clips
with lengths between 30 frames and 150 frames. The batch size is set as 288,
and an AdamW optimizer with a learning rate of 1e-4 is adopted. We randomly
corrupt less than 20% of the Phrases for a sample. The Motion-KP joint space
is trained for 6,000 epochs. While the text-to-motion latent diffusion model
is trained for 3,000 epochs, with the joint space frozen. All experiments are
conducted in 4 NVIDIA RTX 3090 GPUs. More details are provided in the
appendix.

###  6.1 Motion Interpolation

Following Jiang et al. (2023), 50% frames are randomly masked for
interpolation evaluation. FID and Diversity are also evaluated. We adopt MDM
(Tevet et al., 2022b) as the baseline. In Tab. 3, our method provides better
FID. While with additional KPB, the Diversity is increased.

###  6.2 Motion Generation

Settings. We adopt the HumanML3D test set (Guo et al., 2022a) for conventional
text-to-motion evaluation. The evaluation model from Guo et al. (2022a) is
adopted to calculate R-Precision, FID, Diversity, and Multimodality. KPG is
also adopted, with the proposed Accuracy. Also, Diversity is computed as a
reference. We run the evaluation 20 times and report the average metric value.
Details are given in the appendix.

Results on conventional text to motion are shown in Tab. 3. Our method is
competitive without KPB. However, KPB brings a counter-intuitive performance
drop. To evaluate this, we further conduct a user study to make human
volunteers judge the motions instead of a proxy neural network.

| Motion Interpolation | Motion Generation
---|---|---
Methods | FID$\downarrow$ | Diversity$\to$ | R-P@1$\uparrow$ | FID$\downarrow$ | Diversity$\to$ | Multimodality
GT | 0.002 | 9.503 | 0.511 | 0.002 | 9.503 | -
TEMOS (Petrovich et al., 2022) | - | - | 0.424 | 3.734 | 8.973 | 0.368
T2M (Guo et al., 2022a)* | - | - | 0.455 | 1.067 | 9.188 | 2.090
MDM (Tevet et al., 2022b)* | 2.698 | 8.42 | 0.320 | 0.544 | 9.559 | 2.799
TM2T (Guo et al., 2022b)* | - | - | 0.424 | 1.501 | 8.589 | 2.424
MLD (Chen et al., 2023)* | - | - | 0.481 | 0.473 | 9.724 | 2.413
T2M-GPT (Zhang et al., 2023a)* | - | - | 0.492 | 0.141 | 9.722 | 1.831
MotionGPT (Jiang et al., 2023)* | 0.214 | 9.560 | 0.492 | 0.232 | 9.528 | 2.008
Ours* | 0.197 | 9.772 | 0.475 | 0.412 | 10.161 | 2.065
Ours | 0.226 | 10.022 | 0.434 | 0.631 | 10.372 | 2.584

Table 3: Result Comparison of motion interpolation and generation on
HumanML3D. R-P@1 is short for R-Precision@1. * indicates the model is trained
on the HumanML3D train set only.

Our user study is different from previous efforts in two aspects. First,
instead of testing a small set of text prompts (less than 50 in previous works
(Tevet et al., 2022b; Chen et al., 2023)), we randomly select 600 sentences
from the HumanML3D test set. By scaling up, the result is convincing in
reflecting the ability to generate motion for diverse text inputs. Second,
neither asking the volunteers to give a general rating for each sample nor to
choose between different samples, we ask them two questions: 1) Do the motion
and the text match? and 2) Is the motion natural? For Q1, three choices are
given as “No, Partially, Yes”. For Q2, two choices are given as “Yes, No”. In
this way, we explicitly decouple the evaluation of text-to-motion into
semantic consistency and naturalness, corresponding to R-Precision and FID.
For each prompt, we generate one sample considering the annotation cost. We
claim that the models should generate natural text-matching motion most of the
time so that the one-sample setting would not hurt the fidelity of our user
study. 36 volunteers are invited, each reviewing 200 sequences. Thus each
sequence receives 3 user reviews. Also, we compute R-precision@1 of the
generated sequences for reference. MDM (Tevet et al., 2022b), T2M-GPT (Zhang
et al., 2023a), MLD (Chen et al., 2023), and our method are evaluated.

Figure 4:  User study on HumanML3D, with “Y” for Yes and “P” for partially.

Figure 5: User study on KPG, with “Y” for Yes and “P” for partially.

User study results are shown in Fig. 4. Though our method is not superior in
R-Precision, we receive better user reviews, showcasing the efficacy of our
KP-mediated generation strategy. Recent T2M-GPT and MLD present similar
R-Precision, but only T2M-GPT manages to keep a good performance with user
reviews. Moreover, the discrepancy between R-Precision and user reviews is
revealed in both absolute value and trends. More results and analysis are
given in the appendix.

Results on KPG are shown in Tab. 4. KPG is considered an easier task than
conventional text-based motion generation since it is targeted at action
semantics with much less complexity. However, previous methods are not
performing as well as expected. Though we managed to deliver substantial
improvements, the accuracy remains below 60%, which is far from satisfying.
There is a considerable gap between existing methods and ideal motion
generation models.

Furthermore, given the discrepancy between automatic metrics and user study as
shown in Fig. 4, we conducted a similar user study with 100 randomly selected
prompts from KPG involving T2M-GPT and our model. Fig. 5 demonstrates that KP-
inferred Accuracy and user reviews share similar trends. We also calculate
their consistency, showing KP and user study give the same reviews for 84% of
the samples. We believe KPG could thus be a first step towards reliable
automatic motion generation evaluation. More analyses are given in the
appendix.

Methods | Acc.%$\uparrow$ | Diversity
---|---|---
HMDM (Tevet et al., 2022b) | 44.40 | 5.725
MLD (Chen et al., 2023) | 44.76 | 5.901
T2M-GPT (Zhang et al., 2023a) | 47.86 | 6.593
Ours | 52.14 | 6.017

Table 4: Results on Kinematic Prompt Generation.

Methods | Acc.%$\uparrow$ | Diversity
---|---|---
Ours | 52.14 | 6.017
w/o KP mediation | 50.43 | 5.616
Direct mapping | 42.28 | 5.379
w/o Joint KP | 51.03 | 5.765
w/o Joint Pair KP | 48.24 | 5.596
w/o Limb KP | 51.92 | 5.804
w/o Body KP | 52.44 | 5.903

Table 5: Ablation results on KPG.

###  6.3 Visualization

We first present a modification sample in Fig. 6. By modifying KP, we could
edit arbitrary motion at a fine-grained level. Also, We compare generated
samples of T2M-GPT and our methods in Fig. 7. Our method properly responds to
text prompts with constraints on specific body parts. This could be attributed
to KP mediation, which explicitly decomposes the action semantics into
kinematics cues of body parts. Note that T2M-GPT might generate redundant
motion for simple prompts, while our method provides more concise and precise
results. More visualizations are in the appendix.

Figure 6: Our model supports fine-grained modification on motion via
modification on KP.

Figure 7: Visualization of generated samples. Compared to T2M-GPT, our method
provides a better response to prompts with explicit constraints on specific
body parts.

###  6.4 Ablation Studies

Ablation study results on KPG are shown in Tab. 5.

KP mediation. By using our joint space without KP mediation, we still present
a competitive result, showing the efficacy of motion-KP joint space.

Direct mapping. By directly mapping with no KP involved, we present a similar
performance compared to previous methods. It demonstrates the significance of
KP in conveying action semantics.

Different KP sets. We examine the contribution of different KP sets: joint KP
(PP), joint pair KP (PRPP, PDP), limb KP (LAP, LOP), and body KP (GVP). A
leave-one-out style evaluation shows the elimination of joint KP and joint
pair KP results in notable performance degradation, while the influence of the
rest is relatively subtle.

##  7 Discussion

Here, we discuss the limitations and prospects of KP and KP-based
applications. First, KP could be extended beyond its current criteria of sign.
These criteria guarantee objectivity but overlook important kinematic
information like movement amplitude and speed. Also, due to the granularity of
the adopted skeleton, fine-grained kinematic information on fingers is not
well-preserved. The exploration of amplitude/speed/finger-based KP would be a
promising goal to pursue. Second, KPB could be extended to datasets with other
modalities, like 2D pose and egocentric action datasets. Though these
modalities provide incomplete 3D information, we could extract KP that is
credibly accessible across modalities. Third, with the convenient conversion
from KP to text, auxiliary text descriptions could be automatically generated
for motions via KP. Fourth, KPG could be extended by paraphrasing existing
prompts and combining different Phrases.

##  8 Conclusion

In this paper, we proposed an intermediate representation to bridge human
motion and action semantics as the Kinematic Phrase. By focusing on objective
kinematic facts of human motion, KP achieved proper abstraction,
interpretability, and generality. A motion understanding system based on KP
was proposed and proven effective in motion interpolation, modification, and
generation. Moreover, a novel motion generation benchmark Kinematic Prompt
Generation is proposed. We believe that KP has great potential for advancing
motion understanding.

## References

  * Aristidou et al. (2018) Andreas Aristidou, Daniel Cohen-Or, Jessica K. Hodgins, Yiorgos Chrysanthou, and Ariel Shamir.  Deep motifs and motion signatures.  _ACM Trans. Graph._ , 37(6):187:1–187:13, November 2018.  doi: 10.1145/3272127.3275038.  URL http://doi.acm.org/10.1145/3272127.3275038.
  * Athanasiou et al. (2022) Nikos Athanasiou, Mathis Petrovich, Michael J Black, and Gül Varol.  Teach: Temporal action composition for 3d humans.  In _2022 International Conference on 3D Vision (3DV)_ , pp. 414–423. IEEE, 2022.
  * Athanasiou et al. (2023) Nikos Athanasiou, Mathis Petrovich, Michael J. Black, and Gül Varol.  SINC: Spatial composition of 3D human motions for simultaneous action generation.  In _ICCV_ , 2023.
  * Azadi et al. (2023) Samaneh Azadi, Akbar Shah, Thomas Hayes, Devi Parikh, and Sonal Gupta.  Make-an-animation: Large-scale text-conditional 3d human motion generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 15039–15048, October 2023.
  * Bartlett (1997) R. Bartlett.  _Introduction to Sports Biomechanics_.  Introduction to Sports Biomechanics. E & FN Spon, 1997.  ISBN 9780419208402.  URL https://books.google.com.tw/books?id=-6Db8mgxsqQC.
  * Bhatnagar et al. (2022) Bharat Lal Bhatnagar, Xianghui Xie, Ilya Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll.  Behave: Dataset and method for tracking human object interactions.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, jun 2022.
  * Brégier (2021) Romain Brégier.  Deep regression on manifolds: a 3d rotation case study.  In _2021 International Conference on 3D Vision (3DV)_ , pp. 166–174. IEEE, 2021.
  * Cai et al. (2021) Zhongang Cai, Mingyuan Zhang, Jiawei Ren, Chen Wei, Daxuan Ren, Zhengyu Lin, Haiyu Zhao, Lei Yang, and Ziwei Liu.  Playing for 3d human recovery.  _arXiv preprint arXiv:2110.07588_ , 2021.
  * Cai et al. (2022) Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, Fangzhou Hong, Mingyuan Zhang, Chen Change Loy, Lei Yang, and Ziwei Liu.  Humman: Multi-modal 4d human dataset for versatile sensing and modeling.  In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), _Computer Vision – ECCV 2022_ , pp. 557–577, Cham, 2022. Springer Nature Switzerland.
  * Chen et al. (2023) Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu.  Executing your commands via motion diffusion in latent space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18000–18010, 2023.
  * Dabral et al. (2023) Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, and Christian Theobalt.  Mofusion: A framework for denoising-diffusion-based motion synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 9760–9770, June 2023.
  * Delmas et al. (2022) Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Grégory Rogez.  Posescript: 3d human poses from natural language.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VI_ , pp. 346–362. Springer, 2022.
  * Delmas et al. (2023) Ginger Delmas, Philippe Weinzaepfel, Francesc Moreno-Noguer, and Grégory Rogez.  Posefix: Correcting 3d human poses with natural language.  _arXiv preprint arXiv:2309.08480_ , 2023.
  * Fieraru et al. (2020) Mihai Fieraru, Mihai Zanfir, Elisabeta Oneata, Alin-Ionut Popa, Vlad Olaru, and Cristian Sminchisescu.  Three-dimensional reconstruction of human interactions.  In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Fieraru et al. (2021) Mihai Fieraru, Mihai Zanfir, Silviu-Cristian Pirlea, Vlad Olaru, and Cristian Sminchisescu.  Aifit: Automatic 3d human-interpretable feedback models for fitness training.  In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021.
  * Guo et al. (2020) Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng.  Action2motion: Conditioned generation of 3d human motions.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pp. 2021–2029, 2020.
  * Guo et al. (2022a) Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng.  Generating diverse and natural 3d human motions from text.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 5152–5161, June 2022a.
  * Guo et al. (2022b) Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng.  Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts.  In _European Conference on Computer Vision_ , pp. 580–597. Springer, 2022b.
  * Guo et al. (2023) Wen Guo, Yuming Du, Xi Shen, Vincent Lepetit, Xavier Alameda-Pineda, and Francesc Moreno-Noguer.  Back to mlp: A simple baseline for human motion prediction.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 4809–4819, 2023.
  * Hassan et al. (2021) Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael Black.  Stochastic scene-aware motion prediction.  In _Proceedings of the International Conference on Computer Vision 2021_ , October 2021.
  * Hernandez et al. (2019) Alejandro Hernandez, Jurgen Gall, and Francesc Moreno-Noguer.  Human motion prediction via spatio-temporal inpainting.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 7134–7143, 2019.
  * Holden et al. (2020) Daniel Holden, Oussama Kanoun, Maksym Perepichka, and Tiberiu Popa.  Learned motion matching.  _ACM Transactions on Graphics (TOG)_ , 39(4):53–1, 2020.
  * Hong et al. (2022) Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu.  Avatarclip: Zero-shot text-driven generation and animation of 3d avatars.  _arXiv preprint arXiv:2205.08535_ , 2022.
  * Ji et al. (2018) Yanli Ji, Feixiang Xu, Yang Yang, Fumin Shen, Heng Tao Shen, and Wei-Shi Zheng.  A large-scale rgb-d database for arbitrary-view human action recognition.  In _Proceedings of the 26th ACM international Conference on Multimedia_ , pp. 1510–1518, 2018.
  * Jiang et al. (2023) Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen.  Motiongpt: Human motion as a foreign language.  _arXiv preprint arXiv:2306.14795_ , 2023.
  * Karunratanakul et al. (2023) Korrawe Karunratanakul, Konpat Preechakul, Supasorn Suwajanakorn, and Siyu Tang.  Guided motion diffusion for controllable human motion synthesis.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 2151–2162, October 2023.
  * Kong et al. (2023) Hanyang Kong, Kehong Gong, Dongze Lian, Michael Bi Mi, and Xinchao Wang.  Priority-centric human motion generation in discrete latent space.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 14806–14816, October 2023.
  * Koppula & Saxena (2013) Hema Swetha Koppula and Ashutosh Saxena.  Anticipating human activities for reactive robotic response.  In _IROS_ , pp. 2071. Tokyo, 2013.
  * Li et al. (2021a) Ruilong Li, Shan Yang, David A. Ross, and Angjoo Kanazawa.  Learn to dance with aist++: Music conditioned 3d dance generation, 2021a.
  * Li et al. (2021b) Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa.  Ai choreographer: Music conditioned 3d dance generation with aist++.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 13401–13412, 2021b.
  * Lin et al. (2023a) Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang.  Motion-x: A large-scale 3d expressive whole-body human motion dataset.  _arXiv preprint arXiv:2307.00818_ , 2023a.
  * Lin et al. (2023b) Junfan Lin, Jianlong Chang, Lingbo Liu, Guanbin Li, Liang Lin, Qi Tian, and Chang-Wen Chen.  Being comes from not-being: Open-vocabulary text-to-motion generation with wordless training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 23222–23231, June 2023b.
  * Loper et al. (2015) Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  _ACM transactions on graphics (TOG)_ , 34(6):1–16, 2015.
  * Lucas et al. (2022) Thomas Lucas, Fabien Baradel, Philippe Weinzaepfel, and Grégory Rogez.  Posegpt: quantization-based 3d human motion generation and forecasting.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VI_ , pp. 417–435. Springer, 2022.
  * Mahmood et al. (2019) Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black.  Amass: Archive of motion capture as surface shapes.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 5442–5451, 2019.
  * Paden et al. (2016) Brian Paden, Michal Čáp, Sze Zheng Yong, Dmitry Yershov, and Emilio Frazzoli.  A survey of motion planning and control techniques for self-driving urban vehicles.  _IEEE Transactions on intelligent vehicles_ , 1(1):33–55, 2016.
  * Pavlakos et al. (2019) Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3D hands, face, and body from a single image.  In _Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pp. 10975–10985, 2019.
  * Petrovich et al. (2021) Mathis Petrovich, Michael J Black, and Gül Varol.  Action-conditioned 3d human motion synthesis with transformer vae.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 10985–10995, 2021.
  * Petrovich et al. (2022) Mathis Petrovich, Michael J Black, and Gül Varol.  Temos: Generating diverse human motions from textual descriptions.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXII_ , pp. 480–497. Springer, 2022.
  * Petrovich et al. (2023) Mathis Petrovich, Michael J. Black, and Gül Varol.  Tmr: Text-to-motion retrieval using contrastive 3d human motion synthesis.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 9488–9497, October 2023.
  * Plappert et al. (2016) Matthias Plappert, Christian Mandery, and Tamim Asfour.  The kit motion-language dataset.  _Big data_ , 4(4):236–252, 2016.
  * Pons-Moll et al. (2014) Gerard Pons-Moll, David J Fleet, and Bodo Rosenhahn.  Posebits for monocular human pose estimation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 2337–2344, 2014.
  * Punnakkal et al. (2021) Abhinanda R Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J Black.  Babel: bodies, action and behavior with english labels.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 722–731, 2021.
  * Qian et al. (2023) Yijun Qian, Jack Urbanek, Alexander G. Hauptmann, and Jungdam Won.  Breaking the limits of text-conditioned 3d motion synthesis with elaborative descriptions.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 2306–2316, October 2023.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Taheri et al. (2020) Omid Taheri, Nima Ghorbani, Michael J. Black, and Dimitrios Tzionas.  GRAB: A dataset of whole-body human grasping of objects.  In _European Conference on Computer Vision (ECCV)_ , 2020.  URL https://grab.is.tue.mpg.de.
  * Tang et al. (2022) Yansong Tang, Jinpeng Liu, Aoyang Liu, Bin Yang, Wenxun Dai, Yongming Rao, Jiwen Lu, Jie Zhou, and Xiu Li.  Flag3d: A 3d fitness activity dataset with language instruction.  _arXiv preprint arXiv:2212.04638_ , 2022.
  * Tevet et al. (2022a) Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or.  Motionclip: Exposing human motion generation to clip space.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXII_ , pp. 358–374. Springer, 2022a.
  * Tevet et al. (2022b) Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano.  Human motion diffusion model.  _arXiv preprint arXiv:2209.14916_ , 2022b.
  * Van Welbergen et al. (2010) Herwin Van Welbergen, Ben JH Van Basten, Arjan Egges, Zs M Ruttkay, and Mark H Overmars.  Real time animation of virtual humans: a trade-off between naturalness and control.  In _Computer Graphics Forum_ , volume 29, pp. 2530–2554. Wiley Online Library, 2010.
  * von Laban & Lange (1975) R. von Laban and R. Lange.  _Laban’s Principles of Dance and Movement Notation_.  Macdonald & Evans, 1975.  ISBN 9780712116480.  URL https://books.google.com.tw/books?id=-Vr0AAAAMAAJ.
  * Wang et al. (2023) Yin Wang, Zhiying Leng, Frederick W. B. Li, Shun-Cheng Wu, and Xiaohui Liang.  Fg-t2m: Fine-grained text-driven human motion generation via diffusion model.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 22035–22044, October 2023.
  * Xiang et al. (2022) Wangmeng Xiang, Chao Li, Yuxuan Zhou, Biao Wang, and Lei Zhang.  Language supervised training for skeleton-based action recognition.  _arXiv preprint arXiv:2208.05318_ , 2022.
  * Xu et al. (2023) Liang Xu, Ziyang Song, Dongliang Wang, Jing Su, Zhicheng Fang, Chenjing Ding, Weihao Gan, Yichao Yan, Xin Jin, Xiaokang Yang, Wenjun Zeng, and Wei Wu.  Actformer: A gan-based transformer towards general action-conditioned 3d human motion generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 2228–2238, October 2023.
  * Yuan et al. (2023) Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz.  Physdiff: Physics-guided human motion diffusion model.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 16010–16021, October 2023.
  * Zhang et al. (2023a) Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen.  T2m-gpt: Generating human motion from textual descriptions with discrete representations.  _arXiv preprint arXiv:2301.06052_ , 2023a.
  * Zhang et al. (2022) Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu.  Motiondiffuse: Text-driven human motion generation with diffusion model.  _arXiv preprint arXiv:2208.15001_ , 2022.
  * Zhang et al. (2023b) Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu.  Remodiffuse: Retrieval-augmented motion diffusion model.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 364–373, October 2023b.
  * Zhong et al. (2023) Chongyang Zhong, Lei Hu, Zihao Zhang, and Shihong Xia.  Attt2m: Text-driven human motion generation with multi-perspective attention mechanism.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 509–519, October 2023.
  * Zhou et al. (2019) Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li.  On the continuity of rotation representations in neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5745–5753, 2019.
  * Zhou & Wang (2023) Zixiang Zhou and Baoyuan Wang.  Ude: A unified driving engine for human motion generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 5632–5641, June 2023.

## Appendix

##  Appendix A Overview

This supplementary material presents more details and additional results not
included in the main paper due to page limitation. The list of items included
are:

  * •

More details about _kinematic phrase_ in Sec. B.

  * •

More details about _kinematic phrase base_ in Sec. C.

  * •

Our method details in Sec. D.

  * •

Additional experimental details in Sec. E.

##  Appendix B Kinematic Phrase Details

This section lists the details of the six defined types of KP. During
extraction, the indicator is set as zero if it is smaller than 1e-4.

###  B.1 Position Phrase

There are 36 phrases, corresponding to 36 interested $\langle joint,\
reference\ vector\rangle$ pairs like $\langle left\ hand,\ forward\
vector\rangle$. Combinations whose joint and reference vector are directly
correlated are filtered out, like $\langle lefthip,forward\rangle$.

###  B.2 Pairwise Relative Position Phrase

There are 242 phrases corresponding to 242 interested $\langle joint,\
joint,reference\ vector\rangle$ triplets like $\langle left\ hand,\ right\
hand,forward\ vector\rangle$.

###  B.3 Pairwise Distance Phrase

Joint pairs that are connected by human body topology are filtered out, like
hand-elbow and shoulder-hip. There are 81 phrases corresponding to 81
interested $\langle joint,\ joint\rangle$ pairs like $\langle left\ hand,\
right\ hand\rangle$.

###  B.4 Limb Angle Phrase

There are 8 phrases corresponding to 8 interested limbs, listed in the file
KP/lap.txt.

###  B.5 Limb Orientation Phrase

There are 33 phrases corresponding to 33 interested $\langle limb,\ reference\
vector\rangle$ pairs like $\langle left\ shank,\ right\ vector\rangle$.

###  B.6 Global Velocity Phrase

There are 3 phrases corresponding to the velocity direction with respect to
the three reference vectors.

##  Appendix C Kinematic Phrase Base Details

Over 140 K motion sequences are collected to construct the Kinematic Phrase
Base, including 9 M frames (in 30 FPS) with 48 K different sentences, covering
a vocabulary size of 7,418. Here, we illustrate the distribution of the
collected database represented in motion, KP, and text in Fig. 8.

Figure 8: Motion, KP, and text distribution of Kinematic Phrase Base.

Besides, a word cloud visualization of the texts in the database is
illustrated in Fig. 9.

Figure 9: Word cloud visualization of the texts in Kinematic Phrase Base.

##  Appendix D Method Details

###  D.1 Losses for Joint Space Learning

Reconstruction loss $\mathcal{L}_{rec}$ compares the GT with the outputs of
the VAEs. L1 losses are calculated for the motion representation $M,\hat{M}$,
KP $C,\hat{C}$, the skeleton joints $J,\hat{J}$, the down-sampled mesh
vertices $V,\hat{V}$, and the joint accelerations $A,\hat{A}$.

$\mathcal{L}_{rec}=\sum_{\cdot\in\\{m,p,mp\\}}||M_{\cdot}-\hat{M}_{\cdot}||_{1}+||C_{\cdot}-\hat{C}_{\cdot}||_{1}+||J_{\cdot}-\hat{J}_{\cdot}||_{1}+||V_{\cdot}-\hat{V}_{\cdot}||_{1}+||A_{\cdot}-\hat{A}_{\cdot}||_{1}.$
(4)

KL divergence loss $\mathcal{L}_{KL}$ encourages each distribution to be
similar to a normal distribution $\pi=\mathcal{G}(0,I)$ by minimizing the
Kullback-Leibler(KL) divergence between the normal distribution and the
learned motion and KP distributions. The loss is calculated as

$\mathcal{L}_{KL}=KL(\phi_{m},\pi)+KL(\phi_{p},\pi).$ (5)

Distribution alignment loss $\mathcal{L}_{da}$ encourages the distributions of
motion and KP to resemble each other by minimizing the KL divergence between
them. The loss is calculated as

$\mathcal{L}_{da}=KL(\phi_{m},\phi_{p})+KL(\phi_{p},\phi_{m}).$ (6)

Embedding alignment loss $\mathcal{L}_{emb}$ encourages the sampled latent
vectors to be aligned by minimizing their L1 distance. The loss is calculated
as

$\mathcal{L}_{emb}=||z_{m}-z_{p}||_{1}.$ (7)

The overall loss is calculated as

$\mathcal{L}=\lambda_{1}\mathcal{L}_{rec}+\lambda_{2}\mathcal{L}_{KL}+\lambda_{3}\mathcal{L}_{da}+\lambda_{4}\mathcal{L}_{emb},$
(8)

where $\\{\lambda_{i}\\}_{i=1}^{4}$ are weighting coefficients.

##  Appendix E Experiment Details

###  E.1 Implementation Details

The Motion VAE and KP VAE share the same structure: a 4-layer transformer
encoder, a 4-layer transformer decoder, and a fully connected layer for final
outputs. The denoiser adopted for text-to-motion is designed as a 4-layer
transformer decoder. The latent size is set to 256.
$\\{\lambda_{i}\\}_{i=1}^{4}$ are set as 1. The learning rate is decayed at
4,000 epochs for joint space training and at 2,000 epochs for text-to-motion
latent diffusion model training.

###  E.2 Motion Generation Settings

For HumanML3D (Guo et al., 2022a), motion sequences are generated for 10
seconds given a text prompt. For KPG, the models are required to generate 120
frames given a text prompt.

R-Precision is calculated in a similar way to Guo et al. (2022a). For each
generated motion, its text description is mixed with 31 randomly selected
mismatched descriptions from the test set. The cosine distances between the
motion feature and text features are computed. The average accuracy at the
top-1 place is reported.

FID is adopted to measure the divergence between the GT motion distribution
and the generated motion distribution in the latent space.

Diversity measures the variance of the generated motion sequences. It is
calculated as the average latent distance between two randomly sampled
generated motion sets. The set size is set as 300 in this paper.

Multimodality measures the variance of the generated motion sequences within
each text prompt. For each description, two subsets of motion sequences with
the same size are generated, and then the Multimodality is calculated as the
average distance between the two sets of motions in the latent space. The size
of each subset is set as 10 in this paper.

###  E.3 User Study Details

####  E.3.1 User Study Design

As stated in the main text, we adopt a direct Q&A-style user study instead of
a popular preference test or ratings. Here we clarify the reason for this
design choice. First, this design is more suitable in evaluating semantic
consistency, which we identify as categorical instead of continuous at the
sample level. That is, it is hard to tell whether a motion is more raising
left-hand up than another. Instead, there is only whether a motion is raising
left-hand up or not. Therefore, we chose to present a direct question on
semantic consistency. Second, this design explicitly decouples the evaluation
of text-to-motion into semantic consistency and naturalness, corresponding to
R-Precision and FID. When rating motions or choosing between two motions, it
is hard to guarantee the users make choices according to the expected
standard. Therefore, we explicitly ask decoupled binary questions for
decomposition. Third, it helps reduce annotation costs. For preference
testing, the complexity is $O(N^{2})$, while with our user-study protocol, the
complexity is only $O(N)$. In consideration of our primary focus on semantic
consistency, we adopt this protocol. We also admit this protocol is sub-
optimal in naturalness evaluation, which is a continuous factor. We present
the results on naturalness as a reference in the following sections.

####  E.3.2 User Study on Conventional Text-to-Motion

FID = 0.544 R-P@1 = 0.266 | Semantic consistency | Sum
---|---|---
Yes | Partially | No
Naturalness | Yes | 0.40 | 0.18 | 0.10 | 0.68
No | 0.03 | 0.11 | 0.18 | 0.32
Sum | 0.43 | 0.29 | 0.28 | 1

(a) MDM (Tevet et al., 2022b).

FID = 0.212 R-P@1 = 0.473 | Semantic consistency | Sum
---|---|---
Yes | Partially | No
Naturalness | Yes | 0.34 | 0.13 | 0.04 | 0.51
No | 0.10 | 0.14 | 0.25 | 0.49
Sum | 0.44 | 0.27 | 0.29 | 1

(b) MLD (Chen et al., 2023).

FID = 0.141 R-P@1 = 0.292 | Semantic consistency | Sum
---|---|---
Yes | Partially | No
Naturalness | Yes | 0.50 | 0.16 | 0.05 | 0.71
No | 0.06 | 0.08 | 0.15 | 0.29
Sum | 0.56 | 0.24 | 0.20 | 1

(c) T2M-GPT (Zhang et al., 2023a).

FID = 0.631 R-P@1 = 0.274 | Semantic consistency | Sum
---|---|---
Yes | Partially | No
Naturalness | Yes | 0.52 | 0.21 | 0.02 | 0.75
No | 0.05 | 0.06 | 0.14 | 0.25
Sum | 0.57 | 0.27 | 0.16 | 1

(d) Ours.

Table 6: Detailed user study results on HumanML3D.

Detailed user study results on HumanML3D are demonstrated in Tab. 6. As shown,
both FID and R-P@1 are not totally consistent with the user reviews,
indicating these black-box-based metrics might be sub-optimal for motion
generation evaluation. Meanwhile, the four evaluated methods present a similar
positive correlation between semantic consistency and naturalness. Moreover,
it shows that generating natural motions is a little harder than generating
partially semantic-consistent motions, which might be a potential direction to
advance motion generation.

####  E.3.3 User Study on KPG

Accuracy = 50% | Semantic consistency | Sum
---|---|---
Yes | Partially | No
Naturalness | Yes | 0.29 | 0.09 | 0.53 | 0.91
No | 0.04 | 0.01 | 0.04 | 0.09
Sum | 0.33 | 0.10 | 0.57 | 1

(a) T2M-GPT Zhang et al. (2023a).

Accuracy = 54% | Semantic consistency | Sum
---|---|---
Yes | Partially | No
Naturalness | Yes | 0.33 | 0.07 | 0.51 | 0.92
No | 0.04 | 0.01 | 0.04 | 0.08
Sum | 0.37 | 0.08 | 0.55 | 1

(b) Ours.

Table 7: Detailed user study results on KPG.

Detailed user study results on KPG are demonstrated in Tab. 7. Our proposed
Accuracy shares a similar trend with user-reviewed semantic consistency
between the two methods. Both methods receive good naturalness reviews, which
could result from the simple prompt structure of KPG.

| User Reviewed | Sum
---|---|---
Yes | Partially | No
KP-Inferred | Yes | 0.32 | 0.08 | 0.12 | 0.52
No | 0.03 | 0.01 | 0.44 | 0.48
Sum | 0.35 | 0.09 | 0.56 | 1
Table 8: Detailed consistency statistics between KP-inferred Accuracy and
user-reviewed semantic consistency. Figure 10: For KPG, we generate more
concise motion than T2M-GPT (Zhang et al., 2023a).

Furthermore, we provide detailed consistency statistics between KP-inferred
Accuracy and user-reviewed semantic consistency in Tab. 8. Samples generated
from T2M-GPT and our method are included. KP and users provide similar reviews
for over 80% of the samples, showing good consistency. With respect to user
reviews, KP-inferred Accuracy has a higher false positive rate (0.12 / 0.52 =
0.2308) than a false negative rate (0.04 / 0.48 = 0.0833). We find there are
two typical false positive scenarios. First, the generated motion results in
rather small indicators, close to the 1e-4 threshold. KP captures this,
however, it is hard for humans to notice such subtle movements. Second, as
shown in Fig. 10, the generated motions sometimes tend to be redundant
compared to the given prompts. Users might be distracted, overlooking the
targeted semantics. We find this happens more for T2M-GPT generated samples
(in Fig. 7, extra walking motion; in Fig. 10, extra right-hand waving motion),
while our method manages to provide more concise responses. We think this
could partially explain the higher Diversity of T2M-GPT in Tab. 4. For the
first scenario, we think an adaptive threshold w.r.t. the overall motion
intensity would be helpful, since to human perception, the relative amplitude
is usually more important than the absolute amplitude. Also, as stated in Sec.
7, extending KP to amplitude might also help. The second scenario urges us to
rethink the current text-to-motion task setting. For a “matched” motion-text
pair, should the text semantics be a subset of motion semantics, or strictly
match? Also, is it expected to increase diversity by introducing redundant
motions? We identify these questions as interesting points of attack and leave
them for future exploration.


# Bumblebee black holes in light of Event Horizon Telescope observations

Rui Xu Kavli Institute for Astronomy and Astrophysics, Peking University,
Beijing 100871, China  Dicong Liang Kavli Institute for Astronomy and
Astrophysics, Peking University, Beijing 100871, China  Lijing Shao Kavli
Institute for Astronomy and Astrophysics, Peking University, Beijing 100871,
China  National Astronomical Observatories, Chinese Academy of Sciences,
Beijing 100012, China

###### Abstract

We report the existence of novel static spherical black-hole solutions in a
vector-tensor gravitational theory called the bumblebee gravity model which
extends the Einstein-Maxwell theory by allowing the vector to nonminimally
couple to the Ricci curvature tensor. A test of the solutions in the strong-
field regime is performed for the first time using the recent observations of
the supermassive black-hole shadows in the galaxy M87 and the Milky Way from
the Event Horizon Telescope Collaboration. The parameter space is found
largely unexcluded and more experiments are needed to fully bound the theory.

††journal: ApJ

##  1 Introduction

The bumblebee gravity model is a vector-tensor gravitational theory that
serves as an alternative to Einstein’s general relativity (GR). The vector
field in the theory, called the bumblebee field, enters the action the same
way as the electromagnetic (EM) four-potential, only that the bumblebee field
$B_{\mu}$ couples to the Ricci tensor $R_{\mu\nu}$ quadratically and might
acquire a nontrivial background configuration due to a cosmological potential
term $V$. The action of the bumblebee model reads (Kostelecký, 2004)

$\displaystyle S$ $\displaystyle=$ $\displaystyle\int
d^{4}x\sqrt{-g}\left(\frac{1}{2\kappa}R+\frac{\xi}{2\kappa}B^{\mu}B^{\nu}R_{\mu\nu}-\frac{1}{4}B^{\mu\nu}B_{\mu\nu}-V\right)$
(1) $\displaystyle+S_{\rm m},$

where $g$ is the determinant of the metric, $\kappa=8\pi G$ with $G$ being the
gravitational constant and set to unity in the remaining of the work, $R$ is
the Ricci scalar, and $S_{\rm m}$ is the action of conventional matter. The
EM-like field strength is defined as
$B_{\mu\nu}:=D_{\mu}B_{\nu}-D_{\nu}B_{\mu}$ with $D_{\mu}$ being the covariant
derivative. The coupling term $B^{\mu}B^{\nu}R_{\mu\nu}$ controlled by the
constant $\xi$ indicates that when $\xi\neq 0$, the bumblebee vector field
nonminimally couples to the metric tensor, generating more sophisticated
solutions than those in the Einstein-Maxwell theory.

The bumblebee gravity model has been well investigated in the weak-field
regime (Kostelecký, 2004; Bluhm & Kostelecký, 2005; Bluhm, 2006; Bailey &
Kostelecký, 2006; Bluhm et al., 2008a, b; Liang et al., 2022a), and widely
used as an example for theories where spontaneous violation of Lorentz
symmetry occurs as the bumblebee field acquires a certain background
configuration that minimizes the cosmological potential $V$. Because the form
of $V$ is unknown, the practice in studying violation of Lorentz symmetry
usually assumes a constant background configuration of the bumblebee field in
asymptotically flat regions. Then the constant background bumblebee field can
be related to the coefficients for Lorentz-symmetry violation in the popular
framework of the Standard-Model Extension (SME) (Colladay & Kostelecký, 1997,
1998; Kostelecký, 2004), which collects all possible Lorentz-violating terms
at the level of effective field theory (Kostelecký & Mewes, 2009, 2012, 2013;
Kostelecký & Li, 2019) and predicts effects to confront with experiments
(Kostelecký & Russell, 2011; Tasson, 2016), illustrating and complementing
assumptions and conclusions in the general SME framework.

While the study of the bumblebee model in the weak-field regime leads to
stringent constraints on the bumblebee field in asymptotically flat regions,
it becomes more and more essential to obtain strong-field solutions so that
the model as well as Lorentz symmetry can be tested in the strong-field regime
using the state-of-the-art astrophysical observations from gravitational-wave
(GW) observatories (Abbott et al., 2019, 2021a, 2021b) and the black-hole
shadows (Akiyama et al., 2019, 2022a). As a first step to this destination, we
report newly found numerical solutions of black holes in the bumblebee model
and consequently utilize the shadows of the two supermassive black holes
constructed by the Event Horizon Telescope (EHT) Collaboration to constrain
the bumblebee field in the vicinity of the black holes. The numerical
solutions reported here extend the Reissner-Nordström solution as the
bumblebee model can be regarded as an extension of the Einstein-Maxwell
theory. A charge similar to the electric charge can be defined for the
bumblebee field accompanying the black holes. Our results put simultaneous
bounds on the coupling constant $\xi$ and the bumblebee charge of the black
holes for the first time.

##  2 Static spherical black holes in the bumblebee model.

The setup of equations and the numerical method have been described in detail
in Xu et al. (2023). We briefly summarize the results here.

The field equations can be derived from the action in Eq. (1) by variations of
the metric and the bumblebee field. Since we are looking for vacuum solutions
that already minimize the cosmological potential $V$, it then merely plays the
role of a cosmological constant which we will drop in current study of black-
hole solutions. Denoting the background bumblebee field as $b_{\mu}$, the
vacuum field equations turn out to be

$\displaystyle G_{\mu\nu}=\kappa\left(T_{b}\right)_{\mu\nu},$ $\displaystyle
D^{\mu}b_{{\mu\nu}}+\frac{\xi}{\kappa}b^{\mu}R_{\mu\nu}=0,$ (2)

where $G_{\mu\nu}$ is the Einstein tensor and
$b_{\mu\nu}=D_{\mu}b_{\nu}-D_{\nu}b_{\mu}$. The energy-momentum tensor of
$b_{\mu}$ is

$\displaystyle\left(T_{b}\right)_{\mu\nu}$ $\displaystyle=$
$\displaystyle\frac{\xi}{2\kappa}\Big{[}g_{\mu\nu}b^{\alpha}b^{\beta}R_{\alpha\beta}-2b_{\mu}b_{\lambda}R_{\nu}^{\phantom{\nu}\lambda}-2b_{\nu}b_{\lambda}R_{\mu}^{\phantom{\mu}\lambda}$
(3)
$\displaystyle-\Box_{g}(b_{\mu}b_{\nu})-g_{{\mu\nu}}D_{\alpha}D_{\beta}(b^{\alpha}b^{\beta})+D_{\kappa}D_{\mu}\left(b^{\kappa}b_{\nu}\right)$
$\displaystyle+D_{\kappa}D_{\nu}(b_{\mu}b^{\kappa})\Big{]}+b_{\mu\lambda}b_{\nu}^{\phantom{\nu}\lambda}-\frac{1}{4}g_{\mu\nu}b^{\alpha\beta}b_{\alpha\beta},$

with $\Box_{g}=g^{\alpha\beta}D_{\alpha}D_{\beta}$ being the d’Alembertian in
the curved spacetime.

To find static spherical black holes, the metric ansatz

$\displaystyle
ds^{2}=-e^{2\nu}dt^{2}+e^{2\mu}dr^{2}+r^{2}\left(d\theta^{2}+\sin^{2}\theta\,d\phi^{2}\right)$
(4)

can be used. For the background bumblebee field, it can only have the temporal
component $b_{t}$ and the radial component $b_{r}$ to respect the spherical
symmetry. The unknowns $\nu,\,\mu,\,b_{t}$ and $b_{r}$ are functions of the
radial coordinate $r$ and to be solved from the field equations. Similar to
the Maxwell theory, the radial component of the bumblebee field is nondynamic
due to the specific form of the kinetic term in Eq. (1). This is reflected in
the fact that $b_{r}$ and its derivatives can be completely eliminated in the
field equations without raising the orders of derivatives of the other
variables. However, unlike the Einstein-Maxwell theory where the energy-
momentum tensor of the EM field is gauge invariant, the energy-momentum tensor
of the bumblebee field is not. So the nondynamic radial component $b_{r}$, if
nonvanishing, contributes to $\left(T_{b}\right)_{\mu\nu}$, becoming part of
the source for spacetime curvature.

It turns out that there are two families of vacuum solutions corresponding to
vanishing and nonvanishing $b_{r}$ respectively. The first family of
solutions, characterized by $b_{r}=0$, naturally extends the Reissner-
Nordström solution and is what we will test against the EHT observations. The
second family of solutions with nonvanishing $b_{r}$ is characterized by
$R_{rr}=0$, and only exists for $\xi\neq 0$, as it can be seen from the radial
component of the vector field equation in Eq. (2). Let us mention that the
analytic solution found in Casana et al. (2018) belongs to the second family.
An extension for the result of Casana et al. is obtained in Xu (2023), and the
entire second family of solutions, including numerical ones, is studied in Xu
et al. (2023) in detail. The fact that the bumblebee theory possesses two
distinct families of spherical black-hole solutions has not been discussed in
the literature to our best knowledge.

It is interesting to point out that the second family of solutions can be
regarded as a generalization of the Schwarzschild metric due to the fact that
it remarkably includes an analytic solution whose metric is exactly the
Schwarzschild metric

$\displaystyle\nu=-\mu=\frac{1}{2}\ln{\left(1-\frac{2M}{r}\right)},$ (5)

while $b_{t}$ and $b_{r}$ are given by

$\displaystyle b_{t}=\lambda_{0}+\frac{\lambda_{1}}{r},$ $\displaystyle
b_{r}=\pm\bigg{[}\frac{\lambda_{1}^{2}(2r-M)+6\lambda_{0}\lambda_{1}Mr+6\lambda_{0}^{2}M^{2}r}{3M(r-2M)^{2}}$
$\displaystyle\hskip 31.2982pt-\frac{\kappa\lambda_{1}^{2}}{3\xi
M(r-2M)}\bigg{]}^{\frac{1}{2}},$ (6)

with $\lambda_{0}$ and $\lambda_{1}$ being constant. Non-Schwarzschild metric
exists in the second family of solutions, but it happens that the leading-
order effect on the advance of perihelion for an orbit in the spacetime of
these solutions deviates from the GR result regardless of the bumblebee field.
So the Solar-system observations of the planets’ orbits have severely
restricted the difference between the metric of these solutions and the
Schwarzschild metric (Casana et al., 2018). In other words, as long as the
Solar-system constraints are satisfied, the solutions in the second family
converge to the analytic solution given by Eqs. (5) and (6), so the shadow of
a black hole in this family is very close to that of a Schwarzschild black
hole no matter how large $\lambda_{0}$ and $\lambda_{1}$ are. This is why we
are not to test the second family of solutions against the EHT observations.

Focusing on the first family of solutions where $b_{r}=0$, ordinary
differential equations (ODEs) for $\nu,\,\mu$ and $b_{t}$ from Eq. (2) can be
numerically integrated either from a large radius inward or from a given value
of the radius for the event horizon outward. Each black-hole solution in this
family has two independent parameters, maybe chosen as the asymptotic
quantities: the mass $M$ and the bumblebee charge $Q$ of the black hole, or as
the quantities on the event horizon: the radius of the event horizon $r_{h}$
and the limit of $(r-r_{h})\,g_{rr}$ when $r\rightarrow r_{h}$. Practically we
find that the error of the numerical solutions is easier to control when
integrating from the event horizon outward with $r_{h}$ and the limit of
$(r-r_{h})\,g_{rr}$ at $r\rightarrow r_{h}$ specified than integrating from a
large radius inward with the mass and the charge specified. But for the
presentation of the results in this work, let us speak of the mass $M$ and the
charge $Q$ as the two indepent parameters of the black hole as they are
quantities more intuitive than the limit of $(r-r_{h})\,g_{rr}$ at
$r\rightarrow r_{h}$.

Technically, the mass $M$ and the bumblebee charge $Q$ of the black hole are
defined to be proportional to the coefficients of the $1/r$ terms in the
asymptotic expressions for the variables $\mu$ and $b_{t}$ respectively.
Denoting

$\displaystyle\mu=\frac{\mu_{1}}{r}+\frac{\mu_{2}}{r^{2}}+\frac{\mu_{3}}{r^{3}}+\cdots\,,$
$\displaystyle
b_{t}=\lambda_{0}+\frac{\lambda_{1}}{r}+\frac{\lambda_{2}}{r^{2}}+\frac{\lambda_{3}}{r^{3}}+\cdots\,,$
(7)

then $M:=\mu_{1}$ and $Q:=\sqrt{\kappa/2}\,\lambda_{1}$. The other
coefficients $\mu_{2},\,\mu_{3},\,\cdots$ and
$\lambda_{2},\,\lambda_{3},\,\cdots$ in Eq. (7) as well as expansion
coefficients for $\nu$ are related to $\mu_{1}$ and $\lambda_{1}$ (or
equivalently, $M$ and $Q$) by certain recurrence relations that can be derived
from the ODEs (see Eq. (B1) in Xu et al. (2023)). Note that there is no
recurrence relation for the coefficient $\lambda_{0}$, but it depends on
$\mu_{1}$ and $\lambda_{1}$ due to a nontrivial boundary condition for black-
hole solutions, namely that $g_{tt}=-e^{2\nu}$ vanishes on the event horizon.
That is to say, we find in the numerical solutions that $g_{tt}$ is not
guaranteed to vanish when $g_{rr}$ diverges at a finite radius $r_{h}$ which
is supposed to be the event horizon. Solutions with nonvanishing $g_{tt}$ at
the finite radius where $g_{rr}$ diverges and the curvature scalars
$R,\,R^{{\mu\nu}}R_{{\mu\nu}}$ and
$R^{{\alpha\beta\gamma\delta}}R_{{\alpha\beta\gamma\delta}}$ are finite are
discussed in Xu et al. (2023). These solutions are not black holes, so we do
not consider them here.

Figure 1: Radius of the event horizon $r_{h}$ versus bumblebee charge $Q$ of
the black hole. Except for $\xi=2\kappa$, a maximally charged black-hole
solution, indicated by a solid dot on each line, exists for a given coupling
constant $\xi$. Notice that for the green line with $\xi=0.8045\kappa$, the
bumblebee charge $Q$ turns back after reaching its maximum.

By specifying suitable values for $r_{h}$ and the limit of $(r-r_{h})\,g_{rr}$
at $r\rightarrow r_{h}$, we obtain black-hole solutions by integrating the
field equations outward. The mass $M$ and the charge $Q$ for each solution can
then be extracted according to Eq. (7). Using the mass $M$ as the unit, the
radius of the event horizon $r_{h}$ with respect to the charge $Q$ is plotted
in Fig. 1, for different values of the coupling constant $\xi$. As expected,
the numerical solutions recover the Reissner-Nordström solution when $\xi=0$.
What remarkable is that when $\xi=2\kappa$, the metric of the numerical
solutions becomes the Schwarzschild metric, independent of the bumblebee
charge which therefore can be arbitrarily large. In this case, $b_{t}$ also
has an interesting analytic expression, $b_{t}\propto 1-2M/r$. We also notice
that when $\xi>0$, the maximal charge-mass ratio for the black hole is larger
than unity. Finally, let us point out that when $\xi$ is between 0 and about
$1.2\kappa$, two black-hole solutions exist given a value of $Q$ near the
maximal charge (see the green curve with a dashed segment in Fig. 1). We
suspect the one with the smaller $r_{h}$ to be unstable and drop these
solutions in the following analysis though further stability study is needed
to confirm the speculation.

To have a view of the existence domain for the black-hole solutions, we still
use the mass of the black hole as the unit, and generate in Fig. 2 a contour
plot representing different value levels for the radius of the event horizon
on the two-dimensional plane of the coupling constant $\xi$ and the bumblebee
charge $Q$. The boundary of the contour plot indicates the change of the
maximal charge-mass ratio with respect to the coupling constant $\xi$, namely
that black-hole solutions do not exist in the blank region. Notice that
$r_{h}$ does not approach zero at the boundary of the existence domain.
Excluding the suspected unstable solutions corresponding to the dashed segment
in Fig. 1, the minimal radius of the event horizon for all values of $\xi$ is
above $0.7M$. At $\xi=2\kappa$, the metric takes the form of the Schwarzschild
solution with $r_{h}=2M$ regardless of the bumblebee charge $Q$ which can be
arbitrarily large. As $\xi$ goes away from the special value $\xi=2\kappa$,
the maximal charge-mass ratio decreases rapidly.

Figure 2: Contour plot for the radius of the event horizon $r_{h}$ on the
$\xi$-$Q$ plane with the mass of the black hole being the unit of $r_{h}$ and
$Q$. The red dashed lines indicate $\xi=0$ and $\xi=2\kappa$. The uneven
boundary on the right part of the plot is due to numerical difficulties in
finding the maximally charged black-hole solutions when $\xi$ is large.

##  3 Constraints from the EHT observations.

The EHT Collaboration successfully constructed shadow images for the
supermassive black holes in the centers of the galaxy M87 (Akiyama et al.,
2019) and the Milky Way (Akiyama et al., 2022a), providing quantitive
evidences for the scale of black-hole event horizon. When alternatives to GR
predict black holes with different sizes of event horizon, the shadow images
can be used to test and constrain them. As the shadow of a black hole is
casted when the EM waves emitted by matter near the event horizon are absorbed
by the black hole, the exact size and shape of the shadow depend not only on
the size of the event horizon but also on the distribution and the motion of
matter around the black hole. The effect due to matter is complicated, but
generally small when considering its correction to the size of the shadow,
given current observational uncertainties of the masses and distances of the
two supermassive black holes (Akiyama et al., 2019, 2022a). Therefore, we
neglect the effect due to matter in this work but acknowledge that modelling
matter surrounding the black hole is highly demanded for improving the test as
well as explaining image details.

Without concerning the distribution and the motion of matter around a black
hole, the shadow of the black hole is determined by the trajectory of the
light that barely escapes from the light ring near the event horizon. For an
observer far away from the black hole, the radius of the shadow is the
distance between the line of sight and the approaching trajectory of the
light, which is equivalently the impact parameter for the scattering lightlike
geodesic that infinitely orbits around the light ring. In the spacetime
described by the metric in Eq. (4), lightlike geodesics have four-velocity
components

$\displaystyle\frac{dt}{d\lambda}=e^{-2\nu},$
$\displaystyle\frac{dr}{d\lambda}=\pm
e^{-\mu}\sqrt{e^{-2\nu}-\frac{\sigma^{2}}{r^{2}}},$
$\displaystyle\frac{d\theta}{d\lambda}=0,$
$\displaystyle\frac{d\phi}{d\lambda}=\frac{\sigma}{r^{2}},$ (8)

where $\lambda$ is an affine parameter, $\sigma$ is an integral constant, and
spherical symmetry has been used to put the orbit in the equatorial plane. To
see that $\sigma$ is the impact parameter for scattering orbits, we take the
$x$-axis along the velocity direction of the orbit at infinity. Then, as $r$
goes to infinity, we have $\phi\rightarrow 0$, and the impact parameter is

$\displaystyle\lim\limits_{r\rightarrow\infty}r\phi=\lim\limits_{r\rightarrow\infty}r^{2}\times\frac{\big{|}{d\phi}/{d\lambda}\big{|}}{\big{|}{dr}/{d\lambda}\big{|}}=\sigma\lim\limits_{r\rightarrow\infty}e^{\mu+\nu}.$
(9)

The black-hole solutions to be tested here are asymptotically flat, so
$\sigma$ is the impact parameter.111Black-hole solutions in the second family
($b_{r}\neq 0$) can have nonvanishing $\mu$ at infinity though Solar-system
observations are not in favor of such solutions (Casana et al., 2018; Xu et
al., 2023).

The value of the impact parameter $\sigma$ on the light ring gives the radius
of the black-hole shadow. It can be computed using the definition of the light
ring, namely

$\displaystyle\frac{dr}{d\lambda}=0,\quad\frac{d^{2}r}{d\lambda^{2}}=0.$ (10)

The $r$-component of the four-acceleration can be obtained from the geodesic
equations. After some simplification, Eq. (10) gives

$\displaystyle re^{-\nu}=\sigma,\quad r\nu^{\prime}=1,$ (11)

where the prime denotes the derivative with respect to $r$. Our numerical
solutions are interpolated to solve the radius of the light ring, denoted as
$r_{\rm lr}$, from the second equation in Eq. (11). Then using $r_{\rm lr}$,
the first equation in Eq. (11) gives the value of $\sigma$ for the light ring,
denoted as $\sigma_{\rm lr}$, which is the radius of the shadow for
nonspinning black holes. The spin of a black hole causes small distortion to
the shape of the black-hole shadow in GR, correcting deduced parameters by
factors of order unity (Akiyama et al., 2022b, c). We do not know if the same
is true in bumblebee gravity as rigorous solutions of rotating black holes in
the bumblebee theory have not been found to our knowledge. Though it is beyond
the scope of the present work, constructing solutions of rotating black holes
and putting them into test would be worthwhile as the resolution of EHT
improves.222Our investigation aims at providing preliminary constraints on the
bumblebee charge and the coupling constant in the theory in a simplified but
sufficient way. So we focus on the size of the black-hole shadow which is
described by an average diameter of the bright ring modelled by the EHT
Collaboration while do not take into account the asymmetry parameter of the
ring. More accurate constraints can be derived by simultaneously considering
the average diameter and the asymmetry of the observationally modelled black-
hole shadow.

For observational results, we have an angular diameter $d=42\pm 3\,\mu{\rm
as}$ for the shadow of the black hole in M87 (Akiyama et al., 2019), and an
angular diameter $d=51.8\pm 2.3\,\mu{\rm as}$ for the shadow of the black hole
in the Milky Way (Akiyama et al., 2022a). As the angular diameter is related
to the radius $\sigma_{\rm lr}$ by

$\displaystyle d=\frac{2\sigma_{\rm lr}}{D}=\frac{2\sigma_{\rm
lr}}{M}\frac{M}{D},$ (12)

where $M$ and $D$ are the mass and distance of the black hole, the mass-
distance ratio is required to obtain the observational value of $\sigma_{\rm
lr}$ in unit of the black hole mass. For the supermassive black hole in M87,
we adopt $M/D=3.62\pm 0.60\,\mu{\rm as}$ from stellar dynamics (Gebhardt et
al., 2011), and for the supermassive black hole in the Milky Way, we adopt the
average of the results from the Very Large Telescope Interferometer (Abuter et
al., 2022) and Keck (Do et al., 2019) while their difference is used as the
uncertainty, namely $M/D=5.02\pm 0.20\,\mu{\rm as}$.333To test the theory
self-consistently, the mass-distance ratios should also be deduced using the
bumblebee gravity. However, since deducing the mass-distance ratios only
involves using the theory in its weak-field limit where the bumblebee gravity
coincides with GR, we are comfortable with using the standard results in the
literature for the mass-distance ratios. We reckon that using the full theory
would not introduce significant changes. In conclusion, we have

$\displaystyle\frac{\sigma_{\rm lr}}{M}=5.80\pm 1.05$ (13)

for the supermassive black hole in M87, and

$\displaystyle\frac{\sigma_{\rm lr}}{M}=5.16\pm 0.31$ (14)

for the supermassive black hole in the Milky Way. Figure 3 shows the region on
the $\xi$-$Q$ plane for the bumblebee black holes that are consistent with the
results in Eqs. (13) and (14).

Figure 3: Constraints on the $\xi$-$Q$ plane due to the observations of the
shadows for the supermassive black holes in M87 and the Milky Way. The shaded
region is generated by letting numerically calculated $\sigma_{\rm lr}/M$ to
satisfy the upper and lower bounds from Sgr A* in Eq. (14). The dash-dotted
orange line is by satisfying the lower bound in Eq. (13) while the upper bound
in Eq. (13) is trivially satisfied by all the solutions. The solid black line
is the smoothed boundary of the plot in Fig. 2, indicating the existence
domain of the black-hole solutions.

##  4 Conclusions.

In this work, we have presented black-hole solutions in the bumblebee gravity
model described by the action in Eq. (1). The coupling between the bumblebee
vector field and the Ricci tensor enriches the content of the theory, leading
to two families of black-hole solutions. One extends the Reissner-Nordström
solution in the Einstein-Maxwell theory, and the other appears to generalize
the Schwarzschild solution for nonzero coupling constant $\xi$. As the metric
of the solution in the latter family has been constrained very close to the
Schwarzschild metric regardless of the bumblebee field (Xu et al., 2023), we
have only considered testing solutions in the first family in this work.

Using astronomical observations on the shadows and the mass-distance ratios of
two supermassive black holes, we have identified the region for the bumblebee
black holes that are consistent with the observations on the two-dimensional
plane of the coupling constant $\xi$ and the bumblebee charge $Q$ as shown in
Fig. 3. The parameter space is only moderately constrained. We attribute this
to two features in the solutions of the bumblebee model. First, we find that
the metric of the black-hole solution is exactly the Schwarzschild metric
while the bumblebee charge can be arbitrarily large if $\xi=2\kappa$. Thus for
$\xi\sim 2\kappa$, the metric is too close to the Schwarzschild metric for the
bumblebee charge to be constrained by the black-hole shadow observations.
Second, for large $|\xi|$, the maximal bumblebee charge that a black hole can
carry is automatically suppressed in the solution, so the metric is expected
to be close to the Schwarzschild metric anyway. Therefore little parameter
space is excluded for large $|\xi|$ too.

From another point of view, the difficulty in distinguishing a bumblebee black
hole from the Schwarzschild black hole in GR using the observations of black-
hole shadows indicates how capable an alternative to GR the bumblebee model
is. To discriminate between GR and the bumblebee model, state-of-the-art
techniques of multimessenger and multiwavelength astronomy are desirable,
especially observations of GWs and X-rays that encode complementary
information about gravity around black holes in addtion to the radio images
from EHT. Features of GW polarizations in the bumblebee model that are very
different from those in GR have recently been found in Liang et al. (2022a),
and an estimate of constraints using future observations of extrem mass ratio
inspirals (EMRIs) has been carried out in Liang et al. (2022b). Further work
on finding modifications to GWs emitted in coalescences of stellar-mass black
holes would be the next step for putting the bumblebee model into test against
GW observations. Observations of X-rays from accretion disks of astrophysical
black holes also provide exciting opportunities to test black hole spacetime
metric. In fact, the spectral analysis techniques of X-rays are now mature
enough to give even slightly better constraints on alternative black-hole
solutions than those from GWs when elaborate accretion models are set up and
suitable sources are carefully chosen as shown in Tripathi et al. (2021) and
Bambi (2022). As the current package for analyzing X-ray data from black-hole
accretion disks requires analytical spacetime metric solutions (Bambi, 2022),
it would be necessary to incorporate algorithms that can use numerical metric
solutions into the package to test the numerical bumblebee black holes in the
future. In the meanwhile, with the next generation EHT (ngEHT) and space-borne
sub-millimeter interferometries (Blackburn et al., 2019; Gurvits et al.,
2022), sharper black hole images are on the way. A combination of the ngEHT
images with X-ray and GW observations will eventually provide a more complete
landscape for bumblebee-like vector-tensor gravity theories.

We thank Bin Chen and V. Alan Kostelecký for insightful discussions and
comments. This work was supported by the National Natural Science Foundation
of China (11991053, 12147120, 11975027, 11721303), the China Postdoctoral
Science Foundation (2021TQ0018), the National SKA Program of China
(2020SKA0120300), the Max Planck Partner Group Program funded by the Max
Planck Society, and the High-Performance Computing Platform of Peking
University. R.X. is supported by the Boya Postdoctoral Fellowship at Peking
University. Facility: EHT

## References

  * Abbott et al. (2019) Abbott, B. P., et al. 2019, Phys. Rev. X, 9, 031040, doi: 10.1103/PhysRevX.9.031040
  * Abbott et al. (2021a) Abbott, R., et al. 2021a, Phys. Rev. X, 11, 021053, doi: 10.1103/PhysRevX.11.021053
  * Abbott et al. (2021b) —. 2021b.  https://arxiv.org/abs/2111.03606
  * Abuter et al. (2022) Abuter, R., et al. 2022, Astron. Astrophys., 657, L12, doi: 10.1051/0004-6361/202142465
  * Akiyama et al. (2019) Akiyama, K., et al. 2019, Astrophys. J. Lett., 875, L1, doi: 10.3847/2041-8213/ab0ec7
  * Akiyama et al. (2022a) —. 2022a, Astrophys. J. Lett., 930, L12, doi: 10.3847/2041-8213/ac6674
  * Akiyama et al. (2022b) —. 2022b, Astrophys. J. Lett., 930, L15, doi: 10.3847/2041-8213/ac6736
  * Akiyama et al. (2022c) —. 2022c, Astrophys. J. Lett., 930, L16, doi: 10.3847/2041-8213/ac6672
  * Bailey & Kostelecký (2006) Bailey, Q. G., & Kostelecký, V. A. 2006, Phys. Rev. D, 74, 045001, doi: 10.1103/PhysRevD.74.045001
  * Bambi (2022) Bambi, C. 2022.  https://arxiv.org/abs/2210.05322
  * Blackburn et al. (2019) Blackburn, L., et al. 2019.  https://arxiv.org/abs/1909.01411
  * Bluhm (2006) Bluhm, R. 2006, Lect. Notes Phys., 702, 191, doi: 10.1007/3-540-34523-X_8
  * Bluhm et al. (2008a) Bluhm, R., Fung, S.-H., & Kostelecký, V. A. 2008a, Phys. Rev. D, 77, 065020, doi: 10.1103/PhysRevD.77.065020
  * Bluhm et al. (2008b) Bluhm, R., Gagne, N. L., Potting, R., & Vrublevskis, A. 2008b, Phys. Rev. D, 77, 125007, doi: 10.1103/PhysRevD.79.029902
  * Bluhm & Kostelecký (2005) Bluhm, R., & Kostelecký, V. A. 2005, Phys. Rev. D, 71, 065008, doi: 10.1103/PhysRevD.71.065008
  * Casana et al. (2018) Casana, R., Cavalcante, A., Poulis, F. P., & Santos, E. B. 2018, Phys. Rev. D, 97, 104001, doi: 10.1103/PhysRevD.97.104001
  * Colladay & Kostelecký (1997) Colladay, D., & Kostelecký, V. A. 1997, Phys. Rev. D, 55, 6760, doi: 10.1103/PhysRevD.55.6760
  * Colladay & Kostelecký (1998) —. 1998, Phys. Rev. D, 58, 116002, doi: 10.1103/PhysRevD.58.116002
  * Do et al. (2019) Do, T., et al. 2019, Science, 365, 664, doi: 10.1126/science.aav8137
  * Gebhardt et al. (2011) Gebhardt, K., Adams, J., Richstone, D., et al. 2011, Astrophys. J., 729, 119, doi: 10.1088/0004-637X/729/2/119
  * Gurvits et al. (2022) Gurvits, L. I., et al. 2022, Acta Astronaut., 196, 314, doi: 10.1016/j.actaastro.2022.04.020
  * Kostelecký (2004) Kostelecký, V. A. 2004, Phys. Rev. D, 69, 105009, doi: 10.1103/PhysRevD.69.105009
  * Kostelecký & Li (2019) Kostelecký, V. A., & Li, Z. 2019, Phys. Rev. D, 99, 056016, doi: 10.1103/PhysRevD.99.056016
  * Kostelecký & Mewes (2009) Kostelecký, V. A., & Mewes, M. 2009, Phys. Rev. D, 80, 015020, doi: 10.1103/PhysRevD.80.015020
  * Kostelecký & Mewes (2012) —. 2012, Phys. Rev. D, 85, 096005, doi: 10.1103/PhysRevD.85.096005
  * Kostelecký & Mewes (2013) —. 2013, Phys. Rev. D, 88, 096006, doi: 10.1103/PhysRevD.88.096006
  * Kostelecký & Russell (2011) Kostelecký, V. A., & Russell, N. 2011, Rev. Mod. Phys., 83, 11, doi: 10.1103/RevModPhys.83.11
  * Liang et al. (2022a) Liang, D., Xu, R., Lu, X., & Shao, L. 2022a, Phys. Rev. D, 106, 124019, doi: 10.1103/PhysRevD.106.124019
  * Liang et al. (2022b) Liang, D., Xu, R., Mai, Z.-F., & Shao, L. 2022b.  https://arxiv.org/abs/2212.09346
  * Tasson (2016) Tasson, J. D. 2016, Symmetry, 8, 111, doi: 10.3390/sym8110111
  * Tripathi et al. (2021) Tripathi, A., Zhang, Y., Abdikamalov, A. B., et al. 2021, Astrophys. J., 913, 79, doi: 10.3847/1538-4357/abf6cd
  * Xu (2023) Xu, R. 2023, in 9th Meeting on CPT and Lorentz Symmetry.  https://arxiv.org/abs/2301.12666
  * Xu et al. (2023) Xu, R., Liang, D., & Shao, L. 2023, Phys. Rev. D, 107, 024011, doi: 10.1103/PhysRevD.107.024011