# Synthesis of Compositional Animations from Textual Descriptions

Anindita Ghosh Corresponding Author<EMAIL_ADDRESS>German Research
Center for Artificial Intelligence (DFKI)  Saarland Informatics Campus
Noshaba Cheema  German Research Center for Artificial Intelligence (DFKI)
Max-Planck Institute for Informatics  Saarland Informatics Campus  Cennet Oguz
German Research Center for Artificial Intelligence (DFKI)  Christian Theobalt
Max-Planck Institute for Informatics  Saarland Informatics Campus  Philipp
Slusallek  German Research Center for Artificial Intelligence (DFKI)  Saarland
Informatics Campus

###### Abstract

“How can we animate 3D-characters from a movie script or move robots by simply
telling them what we would like them to do?” “How unstructured and complex can
we make a sentence and still generate plausible movements from it?” These are
questions that need to be answered in the long-run, as the field is still in
its infancy. Inspired by these problems, we present a new technique for
generating compositional actions, which handles complex input sentences. Our
output is a 3D pose sequence depicting the actions in the input sentence. We
propose a hierarchical two-stream sequential model to explore a finer joint-
level mapping between natural language sentences and 3D pose sequences
corresponding to the given motion. We learn two manifold representations of
the motion – one each for the upper body and the lower body movements. Our
model can generate plausible pose sequences for short sentences describing
single actions as well as long compositional sentences describing multiple
sequential and superimposed actions. We evaluate our proposed model on the
publicly available KIT Motion-Language Dataset containing 3D pose data with
human-annotated sentences. Experimental results show that our model advances
the state-of-the-art on text-based motion synthesis in objective evaluations
by a margin of 50%. Qualitative evaluations based on a user study indicate
that our synthesized motions are perceived to be the closest to the ground-
truth motion captures for both short and compositional sentences.

Figure 1: Overview of our proposed method to generate motion from complex
natural language sentences.

##  1 Introduction

Manually creating realistic animations of humans performing complex motions is
challenging. Motion synthesis based on textual descriptions substantially
simplifies this task and has a wide range of applications, including language-
based task planning for robotics and virtual assistants [3], designing
instructional videos, creating public safety demonstrations [39], and
visualizing movie scripts [27]. However, mapping natural language text
descriptions to 3D pose sequences for human motions is non-trivial. The input
texts may describe single actions with sequential information (e.g., “a person
walks four steps forward”), or may not correspond to the discrete time steps
of the pose sequences to be generated, in case of superimposed actions (e.g.,
“a person is spinning around while walking”). This necessitates a machine-
level understanding of the syntax and the semantics of the text descriptions
to generate the desired motions [4].

While translating a sentence to a pose sequence, we need to identify the
different parts of speech in the given sentence and how they impact the output
motion. A verb in the sentence describes the type of action, whereas an adverb
may provide information on the direction, place, frequency, and other
circumstances of the denoted action. These need to be mapped into the
generated pose sequence in the correct order, laying out additional challenges
for motion modeling systems. Existing text-to-motion mapping methods can
either generate sentences describing one action only [52] or produce incorrect
results for descriptions of compositional actions [4]. They fail to translate
long-range dependencies and correlations in complex sentences and do not
generalize well to other types of motions outside of locomotion [4]. We
propose a method to handle complex sentences, meaning sentences that describe
a person performing multiple actions either sequentially or simultaneously.
For example, the input sentence “a person is stretching his arms, taking them
down, walking forwards for four steps and raising them again” describes
multiple sequential actions such as raising the arms, taking down the arms,
and walking, as well as the direction and number of steps for the action. To
the best of our knowledge, our method is the first to synthesize plausible
motions from such varieties of complex textual descriptions, which is an
essential next step to improve the practical applicability of text-based
motion synthesis systems. To achieve this goal, we propose a hierarchical,
two-stream, sequential network that synthesizes 3D pose sequences of human
motions by parsing the long-range dependencies of complex sentences,
preserving the essential details of the described motions in the process. Our
output is a sequence of 3D poses generating the animation described in the
sentence (Fig. 1). Our main contributions in this paper are as follows:

Hierarchical joint embedding space. In contrast to [4], we separate our
intermediate pose embeddings into two embeddings, one each for the upper body
and the lower body. We further separate these embeddings hierarchically to
limb embeddings. Our model learns the semantic variations in a sentence
ascribing speed, direction, frequency of motion, and maps them to temporal
pose sequences by decoding the combined embeddings. This results in the
synthesis of pose sequences that correlate strongly with the descriptions
given in the input sentences.

Sequential two-stream network. We introduce a sequential two-stream network
with an autoencoder architecture, with different layers focusing on different
parts of the body, and combine them hierarchically to two representations for
the pose in the manifold space – one for the upper body and the other for the
lower body. This reduces the smoothing of upper body movements (such as wrist
movements for playing violin) in the generated poses and makes the synthesized
motion more robust.

Contextualized BERT embeddings. In contrast to previous approaches [4, 52],
which do not use any contextualized language model, we use the state-of-the-
art BERT model [16] with handpicked word feature embeddings to improve text
understanding.

Additional loss terms and pose discriminator. We add a set of loss terms to
the network training to better condition the learning of the velocity and the
motion manifold [36]. We also add a pose discriminator with an adversarial
loss to further improve the plausibility of the synthesized motions.

Experimental results show that our method outperforms the state-of-the-art
methods of Ahuja et al. [4] and Lin et al. [43] significantly on both the
quantitative metrics we discuss in Section 4.3 and on qualitative evaluations.

##  2 Related Work

This section briefly summarizes prior works in the related areas of data-
driven human motion modeling and text-based motion synthesis.

###  2.1 Human Motion modeling

Data-driven motion synthesis is widely used to generate realistic human motion
for digital human models [33, 31, 17]. Different strategies have been
implemented over the years using temporal convolutional neural networks [14,
40, 10], graph convolution networks [5, 49] and recurrent neural networks [46,
26, 67, 37]. Pose forecasting attempts to generate short [20, 50] and long-
term motions [23, 42, 61] by predicting future sequence of poses given their
history. Prior works encode the observed information of poses to latent
variables and perform predictions based on the latent variables [36, 35].
Holden et al. [34] used a feed-forward network to map high-level parameters to
character movement. Xu et al. [69] proposed a hierarchical style transfer-
based motion generation, where they explored a self-supervised learning method
to decompose a long-range generation task hierarchically. Aristidou et al. [6]
break the whole motion sequences into short-term movements defining motion
words and cluster them to a high-dimensional feature space. Generative
adversarial networks [24] have also gained considerable attention in the field
of unsupervised learning-based motion prediction [8, 38]. Li et al. [41] used
a convolutional discriminator to model human motion sequences to predict
realistic poses. Gui et al. [25] presents the adversarial geometry aware
encoder-decoder (AGED) framework, where two global recurrent discriminators
distinguish the predicted pose from the ground-truth. Cui et al. [15] propose
a generative model for pose modeling based on graph networks and adversarial
learning.

Related work also include pixel-level prediction using human pose as an
intermediate variable [65, 66], locomotion trajectories forecasting [29, 28,
45]. Various audio, speech, and image conditioned forecasting[7] have also
been explored for predicting poses. For instance, [19] explores generating
skeleton pose sequences for dance movements from audio, [9, 68] aims at
predicting future pose sequences from static images. [2] has linked pose
prediction with speech and audio. Takeuchi et al.[60] tackled speech
conditioned forecasting for only the upper body, modeling the non-verbal
behaviors such as head nods, pose switches, hand waving for a character
without providing knowledge on the character’s next movements. [11] rely
solely on the history of poses to predict what kind of motion will follow.

Figure 2: Structure of our hierarchical two-stream model along with pose
discriminator. The model learns a joint embedding for both pose and language.
The embedding has separate representations for the upper body and lower body
movements.

###  2.2 Text-based Motion Synthesis

A subset of prior works have opted to train deep learning models to translate
linguistic instructions to actions for virtual agents [30, 32, 47, 71]. Takano
et al. describe a system that learns a mapping between human motion and word
labels using Hidden Markov Models in [59, 56]. They also used statistical
methods [57, 58] using bigram models for natural languages to generate
motions. Yamada et al. [70] use separate autoencoders for text and animations
with a shared latent space to generate animations from text. Ahn et al. [1]
generates actions from natural language descriptions for video data. However,
their method only applies to upper-body joints (neck, shoulders, elbows, and
wrist joints) with a static root. Recent methods mentioned in [52, 43, 4] used
RNN based sequential networks to map text inputs to motion. Plappert et al.
[52] propose a bidirectional RNN network to map text input to a series of
Gaussian distributions representing the joint angles of the skeleton. However,
their input sequence is encoded into a single one-hot vector that cannot scale
as the input sequence becomes longer. Lin et al. [43] use an autoencoder
architecture to train on mocap data without language descriptions first, and
then use an RNN to map descriptions into these motion representations. Ahuja
et al. [4] learns a joint embedding space for both pose and language using a
curriculum learning approach. Training a model jointly with both pose and
sentence inputs improves the generative power of the model. However, these
methods are limited to synthesize motion from simple sentences. Our model, by
contrast, handles long sentences describing multiple actions.

##  3 Proposed Method

We train our model end-to-end with a hierarchical two-stream pose autoencoder,
a sentence encoder, and pose discriminator as shown in Fig. 2. Our model
learns a joint embedding between the natural language and the poses of the
upper body and the lower body. Our input motion $P=[P_{0},...,P_{T-1}]$ is a
sequence of $T$ poses, where $P_{t}\in\mathbb{R}^{J\times 3}$ is the pose at
$t^{th}$ time step. $J\times 3$ indicate the joints of the skeleton with the
$\left(x,y,z\right)$ coordinates of each joint. Our hierarchical two-stream
pose encoder $(pe)$ encodes the ground truth pose sequence $P$ into two
manifold vectors,

$pe\left(P\right)=(Z^{p}_{ub},Z^{p}_{lb})$ (1)

where $Z^{p}_{ub},Z^{p}_{lb}\in\mathbb{R}^{h}$ represent the features for the
upper body and the lower body, respectively, and $h$ denotes the dimension of
the latent space.

Our input sentence $S=[S_{1},S_{2},...,S_{W}]$ is a sequence of $W$ words
converted to word embeddings $\tilde{S}_{w}$ using the pre-trained BERT model
[16]. $\tilde{S}_{w}\in\mathbb{R}^{K}$ represents the word embedding vector of
the $w^{th}$ word in the sentence and $K$ is the dimension of the word
embedding vector used. Our two-stream sentence encoder $(se)$ encodes the word
embeddings and maps them to the latent space such that we have two latent
vectors,

$se\left(S\right)=(Z^{s}_{ub},Z^{s}_{lb})$ (2)

where $Z^{s}_{ub},Z^{s}_{lb}\in\mathbb{R}^{h}$ represent the sentence
embeddings for the upper body and the lower body, respectively. Using an
appropriate loss (see Section 3.2), we ensure that $(Z^{p}_{ub},Z^{p}_{lb})$
and $(Z^{s}_{ub},Z^{s}_{lb})$ lie close in the joint embedding space and carry
similar information.

Our hierarchical two-stream pose decoder $(de)$ learns to generate poses from
these two manifold vectors. As an initial input, the pose decoder uses the
initial pose $P_{t}$ of time step $t=0$ to generate the pose $\hat{P}_{t}$,
which it uses recursively as input to generate the next pose $\hat{P}_{t+1}$.
$\hat{P}\in\mathbb{R}^{T\times J\times 3}$ denotes a generated pose sequence.
The output of our decoder module is a sequence of $T$ poses
$\hat{P^{p}}\in\mathbb{R}^{T\times J\times 3}$ generated from the pose
embeddings, and $\hat{P^{s}}\in\mathbb{R}^{T\times J\times 3}$ generated from
the language embeddings:

$\displaystyle\hat{P^{p}}$
$\displaystyle=de\left(Z^{p}_{ub},Z^{p}_{lb}\right)$ (3)
$\displaystyle\hat{P^{s}}$
$\displaystyle=de\left(Z^{s}_{ub},Z^{s}_{lb}\right).$ (4)

We use a pose prediction loss term to ensure that $\hat{P^{p}}$ and
$\hat{P^{s}}$ are similar (Section 3.2). $\hat{P}=\hat{P^{s}}$ is our final
output pose sequence for a given sentence.

###  3.1 Network Architecture

The three main modules in our network are the two-stream hierarchical pose
encoder, the two-stream sentence encoder and the two-stream hierarchical pose
decoder. We explain the architecture of all these modules.

####  3.1.1 Two-Stream Hierarchical Pose Encoder

We structure the pose encoder such that it learns features from the different
components of the body. Individual parts are then combined hierarchically. We
decompose the human skeleton into the five major parts - left arm, right arm,
trunk, left leg, and right leg as done in [18]. Our hierarchical pose encoder,
as shown in Fig 2, encodes these five parts using five linear layers with
output dimension $h_{1}$. We combine the trunk representation with that of the
left arm, right arm, left leg, and right leg and pass them through another set
of linear layers to obtain combined representations of (left arm, trunk),
(right arm, trunk), (left leg, trunk), and (right leg, trunk) each of
dimension $h_{2}$ each. Two separate GRUs [12] encode the combined
representation for the arms with trunk and the legs with trunk respectively,
thus creating two manifold representations – one for the upper body
$(Z^{p}_{ub}\in\mathbb{R}^{h})$ and the other for the lower body
$(Z^{p}_{lb}\in\mathbb{R}^{h})$. The output of the GRUs give the two manifold
representations of dimension $h$.

####  3.1.2 Two-Stream Sentence Encoder

To represent the text input, we use the pre-trained large-case model of BERT
[16] as a contextualized language model. It comprises 24 subsequent layers,
each representing different linguistic notions of syntax or semantics [13]. To
find the focused layers on local context (e.g., adverbs of a verb) [62], we
use the attention visualization tool [64] with randomly selected samples of
the KIT Motion Language dataset [51]. Thus, we select the layers $12$
(corresponding to subject(s)), $13$ (adverb(s)), $14$ (verb(s)) and $15$
(prepositional object(s)) and concatenate the hidden states of these layers in
order to represent the corresponding word. Formally,
$\tilde{S}_{w}\in\mathbb{R}^{K}$ represents the word embedding vector of the
$w^{th}$ word in the sentence $S$, and $K$ is the dimension of the word
embedding vector used. Our Sentence encoder $(se)$ uses Long-Short Term Memory
units (LSTMs) [53] to capture the long-range dependencies of a complex
sentence. We input the word embeddings to a two-layer LSTM, which generates
$Z^{s}\in\mathbb{R}^{2h}$, where,

$LSTM\left(\tilde{S}_{w}\right)=Z^{s}=[Z^{s}_{ub},Z^{s}_{lb}]$ (5)

is the latent embedding of the whole sentence, with
$\tilde{S}_{w}=BERT(S_{w})$. We use the first half of this embedding as
$Z^{s}_{ub}\in\mathbb{R}^{h}$ to represent the upper body and the second half
as $Z^{s}_{lb}\in\mathbb{R}^{h}$ to represent the lower body.

####  3.1.3 Two-Stream Hierarchical Pose Decoder

We can conceptually unfold our pose decoder as a series of $T$ hierarchical
decoder units, each constructing the output pose $\hat{P}_{t}$, $\forall
t=0,\dots,T$ time steps in a recurrent fashion by taking in the generated pose
at the corresponding previous time step. We add a residual connection between
the input and the output of the individual decoder units as shown in Fig. 2.
Each decoder unit consists of two GRUs, and a series of linear layers
structured hierarchically. The hierarchical structure of the linear layers in
the decoder unit mirrors that of the pose encoder. Conditioned by the latent
space vector representing the previous frames, the GRUs and the hierarchical
linear layers Hier (as shown in Fig 2) output the reconstructed pose
$\hat{P}_{t+1}$ at the $(t+1)^{th}$ frame given its previous pose
$\hat{P}_{t}$.

###  3.2 Optimizing the Training Procedure

We train our model end-to-end with a hierarchical two-stream pose autoencoder
along with a sentence encoder as shown in Fig. 2. Our model learns a joint
embedding space between the natural language and the poses of the upper body
and the lower body. Our decoder is trained twice in each pass: once with
$(Z^{p}_{ub},Z^{p}_{lb})$ obtained from $pe$ to generate the pose sequence
$\hat{P}^{p}$, and the second time with the $(Z^{s}_{ub},Z^{s}_{lb})$ obtained
from $se$, which generates the pose sequence $\hat{P}=\hat{P}^{s}$.

Loss functions. We use the smooth $\ell_{1}$ loss as a distance metric to
train our model. The smooth $\ell_{1}$ loss is less sensitive to outliers than
the smoother $\ell_{2}$ loss, and more stable than the $\ell_{1}$ loss as it
is differentiable near $x=0$ for all $x\in\mathbb{R}$ [4]. We use the
following losses while training the whole model:

  * •

Pose Prediction loss: It minimizes the difference between the input ground-
truth motion $(P)$ and the predicted motions $\hat{P}=\hat{P}^{s}$ and
$\hat{P}^{p}$. We measure it as,

$L_{R}=\mathcal{L}\left(\hat{P}^{s},P\right)+\mathcal{L}\left(\hat{P}^{p},P\right),$
(6)

where $\mathcal{L}$ denotes the Smooth $\ell_{1}$ Loss between the two terms.

  * •

Manifold reconstruction loss: This encourages a reciprocal mapping between the
generated motions and the manifold representations to improve the manifold
space [36]. For that, we reconstruct the manifold representations from the
generated poses as $\hat{Z}^{p}_{ub}=pe\left(\hat{P}\right)$ and
$\hat{Z}^{p}_{lb}=pe\left(\hat{P}\right)$, and compare them with the manifold
representations obtained from input pose sequence. We compute the loss as,

$L_{M}=\mathcal{L}\left(\hat{Z}^{p}_{ub},Z^{p}_{ub}\right)+\mathcal{L}\left(\hat{Z}^{p}_{lb},Z^{p}_{lb}\right).$
(7)

  * •

Velocity reconstruction loss: We minimize the difference between the velocity
of the reconstructed motion $\left(\hat{P}_{vel}\right)$ and the velocity of
the input motion $\left(P_{vel}\right)$. We compute the velocity of the
$t^{th}$ frame of a pose $P$ as $P_{vel}(t)=P_{(t+1)}-P_{(t)}$. We compute
$L_{V}$ as ,

$L_{V}=\mathcal{L}\left(\hat{P}_{vel},P_{vel}\right).$ (8)

  * •

Embedding similarity loss: We use this loss to ensure that the manifold
representations, $Z^{s}_{ub}$ and $Z^{s}_{lb}$, generated by the sentence
encoder is similar to the manifold representations $Z^{p}_{ub}$ and
$Z^{p}_{lb}$ generated by the pose encoder. We measure it as,

$L_{E}=\mathcal{L}\left(Z^{p}_{ub},Z^{s}_{ub}\right)+\mathcal{L}\left(Z^{p}_{lb},Z^{s}_{lb}\right).$
(9)

  * •

Adversarial loss: We further employ a binary cross-entropy discriminator $D$
to distinguish between the real and generated poses. We compute the
corresponding discriminator and “generator” losses as,

$\displaystyle L_{D}$
$\displaystyle=\mathcal{L}_{2}\left(D\left(\hat{P}\right),0\right)+\mathcal{L}_{2}\left(D\left(P\right),1\right)$
(10) $\displaystyle L_{G}$
$\displaystyle=\mathcal{L}_{2}\left(D\left(\hat{P}\right),1\right),$ (11)

where $\mathcal{L}_{2}$ denotes the Binary Cross Entropy loss, and the
“generator” is the decoder of the auto-encoder.

We train the model end-to-end with the pose autoencoder, the sentence encoder
and the discriminator modules on a weighted sum of these loss terms as,

$\displaystyle\underset{pe,se,de}{\text{min}}\left(L_{R}+\lambda_{M}L_{M}+\lambda_{V}L_{v}+\lambda_{E}L_{E}+\lambda_{G}L_{G}\right)$
$\displaystyle\underset{D}{\text{min}}\left(\lambda_{G}L_{D}\right),$ (12)

where $\lambda_{M}=0.001$, $\lambda_{V}=0.1$, $\lambda_{E}=0.1$ and
$\lambda_{G}=0.001$ are weight parameters, obtained experimentally.

##  4 Experiments

This section describes the dataset we use for our experiments and report the
quantitative and qualitative performance of our method. We also highlight the
benefits of the different components of our method via ablation studies.

###  4.1 Dataset

We evaluate our model on the publicly available KIT Motion-Language Dataset
[51] which consists of $3,911$ recordings of human whole-body motion in MMM
representation [63, 44] with natural language descriptions corresponding to
each motion. There is a total of $6,278$ annotations in natural language, with
each motion recordings having one or multiple annotations describing the task.
The sentences range from describing simple actions such as walking forwards or
waving the hand to describing motions with complicated movements such as
waltzing. Moreover, there are longer, more descriptive sentences describing a
sequence of multiple actions, e.g., “A human walks forwards two steps, pivots
180 degrees and walks two steps back to where they started”. We split the
whole dataset into random samples in the ratio of $0.6$, $0.2$, and $0.2$ for
training, validation, and test sets. For better comparison with the state-of-
the-art [4, 43], we pre-process the given motion data in the same manner as
done in [4, 43]. Following the method of Holden et al. [34], we use the
character’s joint positions with respect to the local coordinate frame and the
character’s trajectory of movement in the global coordinate frame. We have
$J=21$ joints, each having $\left(x,y,z\right)$ coordinates, and a separate
dimension for representing the global trajectory for the root joint. Similar
to [4, 43], we sub-sample the motion sequences to a frequency of $12.5$ Hz
from $100$ Hz.

Figure 3: Comparison of consecutive frames of generated animations of our
method (top row) with Lin et al. [43] (middle row) and JL2P [4] (bottom row)
for the given sentences. Our method generates clear kicking and dancing
motions in contrast to JL2P and Lin et al., which do not show any prominent
movements. The perplexity values of the sentences are according to [51].

###  4.2 Implementation Details

We trained our model for $350$ epochs using Adam Optimizer. Total training
time was approximately $20$ hours using an NVIDIA Tesla V100. The dimensions
of our hidden layers in the hierarchical autoencoder are $h_{1}=32$,
$h_{2}=128$ and $h=512$. We used a batch size of $32$ and a learning rate of
$0.001$ with exponential decay. For training the sentence encoder, we
converted given sentences to word embeddings of dimension $K=4,096$ using
selected layers of the pre-trained BERT-large-case model (details in 3.1.2).
We encoded these embeddings to a dimension of $1024$ through the sentence
encoder, and split it to obtain two manifold representations
$Z^{s}_{ub}\in\mathbb{R}^{h}$ and $Z^{s}_{lb}\in\mathbb{R}^{h}$, each of
dimension $h=512$.

###  4.3 Quantitative Evaluation Metrics

To quantitatively evaluate the correctness of our motion, we use the Average
Position Error (APE). APE measures the average positional difference for a
joint $j$ between the generated poses and the ground-truth pose sequence as,

$APE[j]=\frac{1}{NT}\sum_{n\in N}\sum_{t\in
T}\left\|P_{t}[j]-\hat{P}_{t}[j]\right\|_{2},$ (13)

where $T$ is the total time steps and $N$ is the total number of data in our
test dataset.

Given our setting of natural language descriptions and corresponding free-form
movements, it is naturally difficult to find a quantitative measure that does
justice to both modalities. For example, in a walking setting, sentences that
do not mention any direction correspond to a wider variety of plausible
motions, while specifying a direction narrows the possibilities. To account
for such discrepancies, we separate the APEs between the local joint positions
and the global root trajectory. The former corresponds to the error of the
overall poses, while the latter corresponds to the overall direction and
trajectory of the motion.

However, the average position of each joint simply corresponds to a mean
compared to the dataset. To understand the full statistics of the overall
distribution compared to the dataset, we also compute the Average Variance
Error (AVE), which measures the difference of variances of individual joints
of the generated poses compared to the ground truth poses. We calculate the
variance of an individual joint $j$ for a pose $P$ with $T$ time steps as,

$\sigma[j]=\frac{1}{T-1}\sum_{t\in T}\left(P_{t}[j]-\tilde{P}[j]\right)^{2},$
(14)

where $\tilde{P}[j]$ is the mean pose over $T$ time steps for the joint $j$.
Calculating the variance for all joints of the ground-truth poses and the
generated poses, we use their root mean square error as the AVE metric as
follows:

$AVE[j]=\frac{1}{N}\sum_{n\in N}\left\|\sigma[j]-\hat{\sigma}[j]\right\|_{2},$
(15)

where $\sigma$ refers to the ground-truth pose variance and $\hat{\sigma}$
refers to generated pose variance.

However, even this measure does not account for any information regarding the
sentences or sentence encodings themselves. Therefore, we propose a Content
Encoding Error (CEE), which corresponds to the embedding similarity loss
$L_{E}$ in Eq. 9 by measuring the effectiveness of the embedding space. We
calculate CEE as the difference between manifold representations
$Z^{p}=[Z^{p}_{ub},Z^{p}_{lb}]$ (obtained by encoding the input poses $P$
through the pose encoder $pe$) and the manifold representations
$Z^{s}=[Z^{s}_{ub},Z^{s}_{lb}]$ (obtained by encoding the corresponding input
sentences using the sentence encoder $se$). We write it as,

$CEE(S,P)=\frac{1}{MN}\sum_{n\in N}\sum_{m\in
M}\left\|Z^{s}-Z^{p}\right\|_{2},$ (16)

where $M$ is the number of features in the manifold representation, and $N$ is
the total number of data. The idea is to measure how well the joint embedding
space correlates the latent embeddings of poses with the latent embeddings of
the corresponding sentences.

To further account for style factors in the motion and the sentences, we
propose a Style Encoding Error (SEE). SEE compares a summary statistics of the
sentence embeddings $Z^{s}$ and the pose embeddings $Z^{p}$ to account for
general style information. We compute the Gram matrix [22, 21] $G$ on the
corresponding embeddings:

$\displaystyle G_{s}$ $\displaystyle=Z^{s}\cdot Z^{s^{\top}}$ (17)
$\displaystyle G_{p}$ $\displaystyle=Z^{p}\cdot Z^{p^{\top}}$ (18)

We compute SEE as:

$SEE(S,P)=\frac{1}{MN}\sum_{n\in N}\sum_{m\in
M}\left\|G_{s}-G_{p}\right\|_{2},$ (19)

where $M$ is the number of features in the manifold representation and $N$ is
the total number of data.

Figure 4: Plots showing the APE (left), AVE (middle), CEE and SEE (right) in
mm for our model compared to the JL2P [4], Lin et al. [43]. Dark blue line
denotes our method, grey denotes JL2P and light blue denotes Lin et al.
method. Lower values are better. We see our method improves over the state-of-
the-art by over $50$% on all benchmarks.

###  4.4 Ablation Studies

We compare the performance of our model with the following four ablations
acquired from itself:

  * •

Ablation 1: Two-stream hierarchical model without jointly training the
embedding space (w/o JT). Instead of end-to-end training of the model, we
trained the hierarchical pose encoder and decoder first, using the loss terms
$L_{R}$, $L_{M}$, $L_{V}$, $L_{G}$ and $L_{D}$ (discussed in Section 3.2). We
then trained the model with the sentence encoder and the pose decoder with
$L_{R}$ and $L_{E}$. This indicates that the model is not learning a joint
embedding space for pose and language but learns the embedding space for poses
first and then fine-tunes to map the sentences.

  * •

Ablation 2: Hierarchical model without the two-stream representation (w/o
2-St). We used a single manifold representation for the whole body instead of
separating the upper and lower body and trained the model jointly on language
and pose inputs.

  * •

Ablation 3: Training the hierarchical two-stream model without the extra
losses (w/o Lo). We discarded the additional loss terms introduced in the
paper in Section 3.2 and only used the pose prediction loss $L_{R}$ to train
our model.

  * •

Ablation 4: Using a pre-trained language model instead of selected layers of
BERT (w/o BERT). We used a pre-trained Word2Vec model [48] as done in [4] to
convert the input sentence into word embeddings instead of selecting layers of
BERT as mentioned in Section 3.1.2. This ablation shows how BERT as a
contextualized language model, helped to focus on the local context within a
sentence.

###  4.5 User Study

To evaluate our ablation studies, we conducted a user study to observe the
subjective judgment of the quality of our generated motions compared to the
quality of motions generated from the ablations described in Section 4.4. We
asked 23 participants to rank 14 motion videos from the five methods and from
the ground-truth motion-captures, based on whether the motion corresponds to
the input text, and by the quality and naturalness of the motions. The five
methods include our method and the four ablations of our model – ‘w/o JT’,
‘w/o 2-St’, ‘w/o Lo’, and ‘w/o BERT’. We quantified the user study with two
preference scores – the first one describing if the participants found the
motions to correspond to the input sentence (“yes/no”), and the second one
rating the overall quality of the motion in terms of naturalness (from
$1=$“most natural” to $6=$“least natural”, which we then scaled to 0 and 1 and
inverted). We observe that our method has a preference score of $\sim 40$% in
both cases, second only to the ground truth motion as seen in Fig. 5. 111We
decided to exclude [4] and [43] from the user study, based on overwhelming
feedback from participants that our method beats the state-of-the-art in the
most obvious ways. We provide additional qualitative results for these in the
supplementary material.

##  5 Results and Discussion

We compare our method with the state-of-the-art Joint Language to Pose (JL2P)
method [4], and the proposed approach by Lin et al. [43]. We have used the
pre-trained models for both JL2P and Lin et al.’s approach, provided by Ahuja
et al. [4], to calculate the quantitative results. We computed all the results
on the test dataset.

###  5.1 Objective Evaluation

Fig. 4 shows the improvement of our method compared to JL2P and Lin et al. for
all the metrics discussed in Section 4.3. Our method shows an improvement of
$55.4$% in the mean APE calculated for all local joints compared to JL2P and
by $58.4$% compared to Lin et al. When included with the global trajectory,
our method still shows an improvement of $55.7$% in mean APE compared to JL2P
and an improvement of $58.7$% in mean APE compared to Lin et al. (Fig. 4
left). 222We note that our reported numbers for the state-of-the-art methods
in the APE metric are different from the original paper. However, we were
unable to replicate the numbers in the original paper using the code and the
pre-trained model provided by the authors.

We also observe that high error in the root joint leads to either foot sliding
in the motion or averages out the whole motion. Improvement in the error
values for the root joint indicates high-quality animations without any
artifacts like foot sliding. Furthermore, our method shows closer resemblances
to the variance of the ground truth motion compared to the state-of-the-art
models (Fig. 4 center). Our method has an improvement of $50.4$% in the AVE
over the mean of all joints with global trajectory compared to JL2P, and an
improvement of $50.6$% over the mean of all joints with global trajectory
compared to Lin et al. We provide detailed APE and AVE values of individual
joints in the supplementary material.

We also show improvements of $50$% in the CEE and SEE metrics compared to
JL2P. Compared to Lin et al., we show improvements of $72.3$% and $83.1$% in
the CEE and SEE, respectively (Fig. 4 right). These results show that the
joint embedding space learned by our method can correlate the poses and
corresponding sentences better than the state-of-the-art methods.

Figure 5: Semantic accuracy in percentage denoting how good the motion
visually corresponds to the input sentences (left) and Motion quality in
percentage showing how good the overall quality of motion is in terms of
naturalness (right). Higher value is better. Our method’s score denoted in
red, has highest percentage compared to the ablations.

###  5.2 Qualitative Results

To qualitatively compare our best model against the state-of-the-art methods
[4, 43], we examine the generated motions from all the methods. Fig. 3 shows
two motions with rather high perplexity [51] compared to the average movements
in the dataset. Our method (top row) accurately generates the kicking action
with the correct foot and right arm positions as described in the sentence,
while the benchmark models fail to generate a kick at all (left). Fig. 3
(right) further shows that the Waltz dance is more prominent in our model,
compared to both benchmarks where arm movements seem to be missing completely,
and the skeleton tends to slide than actually step. Fig. 6 shows screenshots
with motions generated from rather complex sentence semantics. Our method (top
row) accurately synthesizes a trajectory that matches the semantics of the
sentence. Although Ahuja et al. [4] generate a circular trajectory (bottom
right), their walking direction does not match the semantics of the sentence,
while Lin et al. [43] fail to generate a circular trajectory at all. Both
methods also cannot synthesize correct turning motions (Fig. 6 left and center
columns).

Figure 6: Comparison of generated animations of our method (top row) with Lin
et al. [43] (middle row) and Ahuja et al. [4] (bottom row) for long sentences
indicating direction and number of steps. Orange cross denotes starting point
and green denotes end point of the motion. Blue line on the plane is the
trajectory and the black dots represent the foot steps. Our method is clearly
able to follow the semantics of the sentences, while the state-of-the-art
fail.

##  6 Limitations, Future Work and Conclusion

We presented a novel framework that advances the state-of-the-art methods on
text-based motion synthesis on qualitative evaluations and several objective
benchmarks. While our model accurately synthesizes superimposed actions it
encountered during training, generalization to novel superimpositions is not
always successful, however. We intend to extend our model into a zero- or few-
shot paradigm [55] such that it generates simultaneous actions from input
sentences without being trained on those specific combinations. We also plan
to experiment with narration-based transcripts that describe long sequences of
step-by-step actions involving multiple people, e.g., narration-based
paragraphs depicting step-by-step movements for performing complex actions
such as dance, work-outs, or professional training videos. To this end, a
different embedding that explicitly models the sequential nature of the task
may be more suitable. However, that may reduce the model’s ability to
synthesize actions not described in an exact sequential manner. Furthermore,
improvements on general motion quality, such as foot sliding, limb
constraints, and biomechanical plausibility, can be improved by introducing
physical constraints [54] to the model.

Being able to model a variety of motions and handle such complex sentence
structures is an essential next step in generating realistic animations for
mixtures of actions in the long-term and improving the practical applicability
of text-based motion synthesis systems. To the best of our knowledge, this is
the first work to achieve this quality of motion synthesis on the benchmark
dataset and is an integral step towards script-based animations.

##### Acknowledgements.

We would like to thank all the participants in our user study, as well as the
XAINES partners for their valuable feedback. This research was funded by the
BMBF grants XAINES (01$|$W20005) and IMPRESS (01$|$S20076), as well as the EU
Horizon 2020 grant Carousel+ (101017779) and an IMPRS-CS Fellowship.

## References

  * [1] H. Ahn, T. Ha, Y. Choi, H. Yoo, and S. Oh.  Text2action: Generative adversarial synthesis from language to action.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5915–5920, 2018.
  * [2] Chaitanya Ahuja.  Coalescing narrative and dialogue for grounded pose forecasting.  In 2019 International Conference on Multimodal Interaction, pages 477–481, 2019.
  * [3] Chaitanya Ahuja, Dong Won Lee, Yukiko I Nakano, and Louis-Philippe Morency.  Style transfer for co-speech gesture animation: A multi-speaker conditional-mixture approach.  In European Conference on Computer Vision, pages 248–265. Springer, 2020.
  * [4] C. Ahuja and L. Morency.  Language2pose: Natural language grounded pose forecasting.  In 2019 International Conference on 3D Vision (3DV), pages 719–728, 2019.
  * [5] Emre Aksan, Manuel Kaufmann, and Otmar Hilliges.  Structured prediction helps 3d human motion modelling.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7144–7153, 2019.
  * [6] Andreas Aristidou, Daniel Cohen-Or, Jessica K Hodgins, Yiorgos Chrysanthou, and Ariel Shamir.  Deep motifs and motion signatures.  ACM Transactions on Graphics (TOG), 37(6):1–13, 2018.
  * [7] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency.  Multimodal machine learning: A survey and taxonomy.  IEEE transactions on pattern analysis and machine intelligence, 41(2):423–443, 2018.
  * [8] Emad Barsoum, John Kender, and Zicheng Liu.  Hp-gan: Probabilistic 3d human motion prediction via gan.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 1418–1427, 2018.
  * [9] Yu-Wei Chao, Jimei Yang, Brian Price, Scott Cohen, and Jia Deng.  Forecasting human dynamics from static images.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 548–556, 2017.
  * [10] Noshaba Cheema, Somayeh Hosseini, Janis Sprenger, Erik Herrmann, Han Du, Klaus Fischer, and Philipp Slusallek.  Dilated temporal fully-convolutional network for semantic segmentation of motion capture data.  arXiv preprint arXiv:1806.09174, 2018.
  * [11] Hsu-kuang Chiu, Ehsan Adeli, Borui Wang, De-An Huang, and Juan Carlos Niebles.  Action-agnostic human pose forecasting.  In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1423–1432. IEEE, 2019.
  * [12] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio.  On the properties of neural machine translation: Encoder-decoder approaches.  arXiv preprint arXiv:1409.1259, 2014.
  * [13] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning.  What does bert look at? an analysis of bert’s attention.  In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, 2019.
  * [14] Qiongjie Cui, Huaijiang Sun, Yue Kong, Xiaoqian Zhang, and Yanmeng Li.  Efficient human motion prediction using temporal convolutional generative adversarial network.  Information Sciences, 545:427–447, 2021.
  * [15] Qiongjie Cui, Huaijiang Sun, and Fei Yang.  Learning dynamic relationships for 3d human motion prediction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6519–6527, 2020.
  * [16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [17] Han Du, Erik Herrmann, Janis Sprenger, Noshaba Cheema, Somayeh Hosseini, Klaus Fischer, and Philipp Slusallek.  Stylistic locomotion modeling with conditional variational autoencoder.  In Eurographics (Short Papers), pages 9–12, 2019.
  * [18] Yong Du, Wei Wang, and Liang Wang.  Hierarchical recurrent neural network for skeleton based action recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1110–1118, 2015.
  * [19] Joao P Ferreira, Thiago M Coutinho, Thiago L Gomes, José F Neto, Rafael Azevedo, Renato Martins, and Erickson R Nascimento.  Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio.  Computers & Graphics, 94:11–21, 2020.
  * [20] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik.  Recurrent network models for human dynamics.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4346–4354, 2015.
  * [21] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  Texture synthesis using convolutional neural networks.  arXiv preprint arXiv:1505.07376, 2015.
  * [22] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016.
  * [23] Partha Ghosh, Jie Song, Emre Aksan, and Otmar Hilliges.  Learning human motion models for long-term predictions.  In 2017 International Conference on 3D Vision (3DV), pages 458–466. IEEE, 2017.
  * [24] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  arXiv preprint arXiv:1406.2661, 2014.
  * [25] Liang-Yan Gui, Yu-Xiong Wang, Xiaodan Liang, and José MF Moura.  Adversarial geometry-aware human motion prediction.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 786–803, 2018.
  * [26] Ikhsanul Habibie, Daniel Holden, Jonathan Schwarz, Joe Yearsley, and Taku Komura.  A recurrent variational autoencoder for human motion synthesis.  In 28th British Machine Vision Conference, 2017.
  * [27] Eva Hanser, Paul Mc Kevitt, Tom Lunney, and Joan Condell.  Scenemaker: Intelligent multimodal visualisation of natural language scripts.  In Irish Conference on Artificial Intelligence and Cognitive Science, pages 144–153. Springer, 2009.
  * [28] Irtiza Hasan, Francesco Setti, Theodore Tsesmelis, Vasileios Belagiannis, Sikandar Amin, Alessio Del Bue, Marco Cristani, and Fabio Galasso.  Forecasting people trajectories and head poses by jointly reasoning on tracklets and vislets.  IEEE transactions on pattern analysis and machine intelligence, 2019\.
  * [29] Irtiza Hasan, Francesco Setti, Theodore Tsesmelis, Alessio Del Bue, Marco Cristani, and Fabio Galasso.  ” seeing is believing”: Pedestrian trajectory forecasting using visual frustum of attention.  In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1178–1185. IEEE, 2018.
  * [30] Jun Hatori, Yuta Kikuchi, Sosuke Kobayashi, Kuniyuki Takahashi, Yuta Tsuboi, Yuya Unno, Wilson Ko, and Jethro Tan.  Interactively picking real-world objects with unconstrained spoken language instructions.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3774–3781. IEEE, 2018.
  * [31] Gustav Eje Henter, Simon Alexanderson, and Jonas Beskow.  Moglow: Probabilistic and controllable motion synthesis using normalising flows.  ACM Transactions on Graphics (TOG), 39(6):1–14, 2020.
  * [32] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al.  Grounded language learning in a simulated 3d world.  arXiv preprint arXiv:1706.06551, 2017.
  * [33] Daniel Holden, Taku Komura, and Jun Saito.  Phase-functioned neural networks for character control.  ACM Transactions on Graphics (TOG), 36(4):1–13, 2017.
  * [34] Daniel Holden, Jun Saito, and Taku Komura.  A deep learning framework for character motion synthesis and editing.  ACM Transactions on Graphics (TOG), 35(4):1–11, 2016.
  * [35] Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce.  Learning motion manifolds with convolutional autoencoders.  In SIGGRAPH Asia 2015 Technical Briefs, pages 1–4. 2015.
  * [36] Deok-Kyeong Jang and Sung-Hee Lee.  Constructing human motion manifold with sequential networks.  In Computer Graphics Forum, volume 39, pages 314–324. Wiley Online Library, 2020.
  * [37] Jogendra Nath Kundu, Himanshu Buckchash, Priyanka Mandikal, Anirudh Jamkhandi, Venkatesh Babu RADHAKRISHNAN, et al.  Cross-conditioned recurrent networks for long-term synthesis of inter-person human motion interactions.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2724–2733, 2020.
  * [38] Jogendra Nath Kundu, Maharshi Gor, and R Venkatesh Babu.  Bihmp-gan: Bidirectional 3d human motion prediction gan.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 8553–8560, 2019.
  * [39] Jasper LaFortune and Kristen L Macuga.  Learning movements from a virtual instructor: Effects of spatial orientation, immersion, and expertise.  Journal of Experimental Psychology: Applied, 24(4):521, 2018.
  * [40] Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager.  Temporal convolutional networks for action segmentation and detection.  In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 156–165, 2017.
  * [41] Chen Li, Zhen Zhang, Wee Sun Lee, and Gim Hee Lee.  Convolutional sequence to sequence model for human dynamics.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5226–5234, 2018.
  * [42] Zimo Li, Yi Zhou, Shuangjiu Xiao, Chong He, Zeng Huang, and Hao Li.  Auto-conditioned recurrent networks for extended complex human motion synthesis.  arXiv preprint arXiv:1707.05363, 2017.
  * [43] Angela S Lin, Lemeng Wu, Rodolfo Corona, Kevin Tai, Qixing Huang, and Raymond J Mooney.  Generating animated videos of human activities from natural language descriptions.  Learning, 2018:2, 2018.
  * [44] Christian Mandery, Ömer Terlemez, Martin Do, Nikolaus Vahrenkamp, and Tamim Asfour.  Unifying representations and large-scale whole-body motion databases for studying human motion.  IEEE Transactions on Robotics, 32(4):796–809, 2016.
  * [45] Karttikeya Mangalam, Ehsan Adeli, Kuan-Hui Lee, Adrien Gaidon, and Juan Carlos Niebles.  Disentangling human dynamics for pedestrian locomotion forecasting with noisy supervision.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2784–2793, 2020.
  * [46] Julieta Martinez, Michael J Black, and Javier Romero.  On human motion prediction using recurrent neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2891–2900, 2017.
  * [47] Hongyuan Mei, Mohit Bansal, and Matthew Walter.  Listen, attend, and walk: Neural mapping of navigational instructions to action sequences.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
  * [48] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean.  Distributed representations of words and phrases and their compositionality.  arXiv preprint arXiv:1310.4546, 2013.
  * [49] Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel.  Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14424–14432, 2020.
  * [50] Dario Pavllo, David Grangier, and Michael Auli.  Quaternet: A quaternion-based recurrent model for human motion.  arXiv preprint arXiv:1805.06485, 2018.
  * [51] Matthias Plappert, Christian Mandery, and Tamim Asfour.  The KIT motion-language dataset.  Big Data, 4(4):236–252, dec 2016.
  * [52] Matthias Plappert, Christian Mandery, and Tamim Asfour.  Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks.  Robotics and Autonomous Systems, 109:13 – 26, 2018.
  * [53] Jürgen Schmidhuber and Sepp Hochreiter.  Long short-term memory.  Neural Comput, 9(8):1735–1780, 1997.
  * [54] Soshi Shimada, Vladislav Golyanik, Weipeng Xu, and Christian Theobalt.  Physcap: Physically plausible monocular 3d motion capture in real time.  ACM Transactions on Graphics (TOG), 39(6):1–16, 2020.
  * [55] Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D Manning, and Andrew Y Ng.  Zero-shot learning through cross-modal transfer.  arXiv preprint arXiv:1301.3666, 2013.
  * [56] Wataru Takano, Dana Kulic, and Yoshihiko Nakamura.  Interactive topology formation of linguistic space and motion space.  In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1416–1422. IEEE, 2007.
  * [57] Wataru Takano and Yoshihiko Nakamura.  Bigram-based natural language model and statistical motion symbol model for scalable language of humanoid robots.  In 2012 IEEE International Conference on Robotics and Automation, pages 1232–1237. IEEE, 2012.
  * [58] Wataru Takano and Yoshihiko Nakamura.  Statistical mutual conversion between whole body motion primitives and linguistic sentences for human motions.  The International Journal of Robotics Research, 34(10):1314–1328, 2015.
  * [59] Wataru Takano and Yoshihiko Nakamura.  Symbolically structured database for human whole body motions based on association between motion symbols and motion words.  Robotics and Autonomous Systems, 66:75–85, 2015.
  * [60] Kenta Takeuchi, Dai Hasegawa, Shinichi Shirakawa, Naoshi Kaneko, Hiroshi Sakuta, and Kazuhiko Sumi.  Speech-to-gesture generation: A challenge in deep learning approach with bi-directional lstm.  In Proceedings of the 5th International Conference on Human Agent Interaction, pages 365–369, 2017.
  * [61] Yongyi Tang, Lin Ma, Wei Liu, and Weishi Zheng.  Long-term human motion prediction by modeling motion context and enhancing motion dynamic.  arXiv preprint arXiv:1805.02513, 2018.
  * [62] Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al.  What do you learn from context? probing for sentence structure in contextualized word representations.  arXiv preprint arXiv:1905.06316, 2019.
  * [63] Ömer Terlemez, Stefan Ulbrich, Christian Mandery, Martin Do, Nikolaus Vahrenkamp, and Tamim Asfour.  Master motor map (mmm)—framework and toolkit for capturing, representing, and reproducing human motion on humanoid robots.  In 2014 IEEE-RAS International Conference on Humanoid Robots, pages 894–901. IEEE, 2014.
  * [64] Jesse Vig.  A multiscale visualization of attention in the transformer model.  arXiv preprint arXiv:1906.05714, 2019.
  * [65] Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee.  Learning to generate long-term future via hierarchical prediction.  In international conference on machine learning, pages 3560–3569. PMLR, 2017.
  * [66] Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert.  The pose knows: Video forecasting by generating pose futures.  In Proceedings of the IEEE international conference on computer vision, pages 3332–3341, 2017.
  * [67] Zhiyong Wang, Jinxiang Chai, and Shihong Xia.  Combining recurrent neural networks and adversarial training for human motion synthesis and control.  IEEE transactions on visualization and computer graphics, 27(1):14–28, 2019.
  * [68] Erwin Wu and Hideki Koike.  Real-time human motion forecasting using a rgb camera.  In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 1575–1577. IEEE, 2019.
  * [69] Jingwei Xu, Huazhe Xu, Bingbing Ni, Xiaokang Yang, Xiaolong Wang, and Trevor Darrell.  Hierarchical style-based networks for motion synthesis.  arXiv preprint arXiv:2008.10162, 2020.
  * [70] Tatsuro Yamada, Hiroyuki Matsunaga, and Tetsuya Ogata.  Paired recurrent autoencoders for bidirectional translation between robot actions and linguistic descriptions.  IEEE Robotics and Automation Letters, 3(4):3441–3448, 2018.
  * [71] Tatsuro Yamada, Shingo Murata, Hiroaki Arie, and Tetsuya Ogata.  Dynamical integration of language and behavior in a recurrent neural network for human–robot interaction.  Frontiers in neurorobotics, 10:5, 2016.

Table 1: Average Positional Error (APE) in mm for our model compared to the JL2P [4], Lin et al. [43], and four ablations of our method described in Section 4.4 of our paper. Although the over all APE is lower for our ablation studies, we find the overall motion quality to be poorer than our final method due to larger errors in the root. Please refer to Section 5.1 of our paper for details. | JL2P | Lin et al. | w/o BERT | w/o JT | w/o 2-St | w/o Lo | Ours
---|---|---|---|---|---|---|---
Trajectory | 4.12 | 4.52 | 1.21 | 1.27 | 1.22 | 1.23 | 1.22
Root | 7.28 | 7.78 | 3.23 | 3.50 | 3.22 | 3.23 | 3.21
Torso | 13.18 | 14.93 | 5.84 | 5.71 | 5.71 | 5.91 | 5.90
Pelvis | 14.92 | 16.10 | 6.49 | 6.54 | 6.52 | 6.67 | 6.60
Neck | 33.01 | 36.03 | 14.88 | 14.50 | 14.69 | 15.04 | 15.01
Left Arm | 37.37 | 41.71 | 16.54 | 16.79 | 16.09 | 16.79 | 16.94
Right Arm | 37.91 | 42.33 | 16.41 | 16.56 | 15.81 | 16.25 | 16.40
Left Hip | 13.50 | 14.33 | 6.02 | 6.12 | 6.14 | 6.18 | 6.21
Right Hip | 13.39 | 14.05 | 6.00 | 6.15 | 6.15 | 6.20 | 6.22
Left Foot | 38.38 | 38.84 | 16.78 | 16.63 | 16.84 | 16.25 | 16.97
Right Foot | 39.66 | 40.31 | 17.12 | 17.15 | 17.24 | 16.78 | 17.22
Mean w/o trajectory | 24.86 | 26.64 | 10.93 | 10.96 | 10.84 | 10.93 | 11.07
Mean | 22.97 | 24.63 | 10.04 | 10.08 | 9.97 | 10.05 | 10.17
Table 2: Average Variance Error (AVE) for our model compared to the JL2P [4], Lin et al. [43], and the four ablations of our method described in Section 4.4 of our paper. Our method has the lowest AVE for the root joint as well as the mean of all the joints with and without the global trajectory. | JL2P | Lin et al. | w/o BERT | w/o JT | w/o 2-St | w/o Lo | Ours
---|---|---|---|---|---|---|---
Trajectory | 18.55 | 19.00 | 10.87 | 10.52 | 11.20 | 9.75 | 10.29
Root | 4.70 | 5.46 | 2.45 | 2.42 | 2.32 | 2.30 | 2.19
Torso | 21.44 | 22.61 | 12.65 | 12.20 | 13.22 | 11.85 | 11.87
Pelvis | 23.79 | 24.51 | 13.66 | 13.25 | 13.99 | 12.73 | 12.58
Neck | 45.05 | 36.03 | 26.24 | 25.26 | 27.37 | 24.78 | 24.65
Left Arm | 32.66 | 41.71 | 16.59 | 16.42 | 16.86 | 15.66 | 15.20
Right Arm | 29.15 | 42.34 | 15.18 | 14.54 | 15.05 | 14.31 | 13.95
Left Hip | 27.79 | 28.73 | 16.01 | 15.45 | 15.82 | 14.35 | 14.71
Right Hip | 26.73 | 27.05 | 14.46 | 14.13 | 14.92 | 13.31 | 13.40
Left Foot | 48.34 | 38.84 | 24.63 | 24.03 | 23.67 | 22.27 | 21.57
Right Foot | 47.23 | 40.31 | 23.04 | 23.10 | 22.80 | 20.72 | 20.87
Mean w/o Trajectory | 30.69 | 30.75 | 16.49 | 16.08 | 16.60 | 15.22 | 15.09
Mean | 29.58 | 29.69 | 15.98 | 15.57 | 16.11 | 14.73 | 14.66

## Appendix: More Results on Quantitative Evaluation Metrics

We show the average positional error (APE) values for individual joints in
Table 1. We compare our method with the two state-of-the-art methods [4, 43]
and also with the four ablations of our method: ‘w/o BERT’, ‘w/o JT’, ‘w/o
2-St’, ‘w/o Lo’, as described in Section $4.4$ of our paper. We observe that
high error in the root joint leads to either foot sliding in the motion or
averages out the whole motion. Improvement in the error values for the root
joint indicates high-quality animations without any such artifacts. When
compared to the ablations of our model, we find that the APE calculated over
the mean of all the joints with the global trajectory is marginally better for
the ablations compared to our method (best for the ablation ‘w/o 2-St’ showing
an improvement of $1.96$% over our method). This is because the motions get
averaged out in the ablations, bringing the joint positions closer to the mean
but reducing the relevant joint movements. However, our method has the lowest
APE for the root joint, implying that the overall motion quality is better.
Using the additional metric of the average variance error (AVE) for evaluating
the variability of the motions further shows that the joint movements are
reduced in the ablations. Our method has the lowest AVE for the root joint as
well as the mean of all the joints with and without the global trajectory, as
shown in Table 2. Our method also performs the best in terms of the content
encoding error (CEE) and the style encoding error (SEE) compared to the
ablations and the state-of-the-art methods as seen in Table 3.

Table 3: Content Encoding Error (CEE) and Style Encoding Error (SEE) for our model compared to the JL2P [4], Lin et al. [43], and the four ablations of our method described in Section 4.4 of our paper. Lower values are better. Our method has the lowest CEE and SEE. Method | JL2P | Lin et al. | w/o BERT | w/o JT | w/o 2-St | w/o Lo | Ours
---|---|---|---|---|---|---|---
CEE | 1.06 | 1.92 | 1.10 | 0.99 | 0.67 | 1.04 | 0.53
SEE | 0.38 | 1.13 | 0.80 | 0.76 | 0.46 | 0.77 | 0.19


Nodes on quintic spectrahedra
Primary: 14J17, 14M12, 14P10, 14Q10, 52A15.
Secondary: 14P25, 90C22.

# Nodes on quintic spectrahedra

Taylor Brysiewicz<EMAIL_ADDRESS>Max Planck Institute for
Mathematics in the Sciences, Leipzig, Germany
Khazhgali Kozhasov<EMAIL_ADDRESS>Technische Universität
Braunschweig, Germany
Mario Kummer<EMAIL_ADDRESS>Technische Universität Dresden,
Germany

(2020/09/08)

###### Abstract

We classify transversal quintic spectrahedra by the location of $20$ nodes on
the respective real determinantal surface of degree $5$. We identify $65$
classes of such surfaces and find an explicit representative in each of them.

###### keywords:

Symmetroid, spectrahedron, real determinantal surface

##  1 Introduction

Spectrahedra are _nonlinear_ generalizations of convex polytopes. They are
convex sets that are obtained as intersections of the _cone
$\mathbb{S}^{n}_{\succcurlyeq}\subset\mathbb{S}^{n}$ of positive semidefinite
matrices_ with a real affine linear subspace
$\mathcal{L}\subset\mathbb{S}^{n}$ of symmetric matrices. Spectrahedra
naturally appear in optimization: they are feasibility regions for
_semidefinite programs_ in the same way as polyhedra are feasibility regions
for _linear programs_. Apart from optimization, spectrahedra have appearances
in algebraic geometry, convex geometry, statistics, and combinatorics. It is
conjectured that the class of spectrahedra is so rich that it contains
_hyperbolicity cones_ of hyperbolic polynomials [7]. In general, it is of
interest to study spectrahedra in any dimension, however, even spectrahedra in
dimension $3$ are far from being completely understood. In the present work we
are mainly interested in $3$-dimensional spectrahedra and work in the setting
of projective geometry. In particular, we assume that
$\mathcal{L}\subset\mathbb{S}^{n}$ is a real linear subspace of dimension $4$,
whose corresponding spectrahedron is
$\mathbb{P}(\mathcal{L}\cap\mathbb{S}^{n}_{\succcurlyeq})$.

Spectrahedra are intimately related to algebraic surfaces called
_symmetroids_. A _symmetroid_ is a surface in projective $3$-space whose
defining polynomial is the determinant of a symmetric matrix of linear forms

$\displaystyle A(t,\bar{x})\ =\ tA_{0}+x_{1}A_{1}+x_{2}A_{2}+x_{3}A_{3},\quad
A_{0},A_{1},A_{2},A_{3}\in\mathbb{S}^{n}.$ (1)

The spectrahedron associated with (1) is the set
$\\{(t,\bar{x})\,:\,A(t,\bar{x})\succeq 0\\}$. If this has nonempty interior,
then $\det(A(t,\bar{x}))$ is a _spectrahedral symmetroid_ containing the
algebraic boundary of the spectrahedron.

A generic symmetroid of degree $n$ carries ${n+1\choose 3}$ _nodal
singularities_ (points of multiplicity $2$) which are represented by matrices
$A(t,\bar{x})$ of corank $2$. Any symmetroid with this property will be called
_transversal_. If a transversal symmetroid is spectrahedral, then
$\rho\leq{n+1\choose 3}$ many of its singular points are real, amongst which
$\sigma\leq\rho$ lie on the Euclidean boundary of the associated
spectrahedron. Characterizing possible _combinatorial types_ $(\rho,\sigma)$
of spectrahedral symmetroids of a given degree $n$ is a natural problem of
real algebraic geometry. Combinatorial types of generic quartic ($n=4$)
spectrahedral symmetroids were classified in [3], see also [8] for an
alternative proof and explicit examples of symmetroids realizing each
combinatorial type.

The main result of the present work is to provide a classification of
combinatorial types of transversal quintic ($n=5$) spectrahedral symmetroids.

######  Theorem 1.

There exists a transversal quintic spectrahedral symmetroid of type
$(\rho,\sigma)$ if and only if $0\leq\sigma\leq\rho\leq 20$, both $\rho$ and
$\sigma$ are even, and $2\leq\rho$.

In Section 2 we determine restrictions that a combinatorial type of a
spectrahedral symmetroid of any given degree $n$ has to satisfy (see Corollary
6). For $n=5$ these are conditions given in Theorem 1. In Section 4, using a
_hill-climbing algorithm_ we find quintic symmetroids realizing each of the
$65$ possible types explicitly and certify the findings a posteriori. The code
and files for reproducing the certification may be found at
https://mathrepo.mis.mpg.de/.

In Section 3 we study real quintic symmetroids with _tetrahedral_ and
_3-prismatic_ symmetries. In both cases singular points split up into orbits
of the action of the corresponding subgroup of the symmetric group. In
particular, if an orbit is real, it either entirely lies on the topological
boundary of the associated spectrahedron or is disjoint from it. Transversal
symmetroids with tetrahedral symmetry form a one-parameter family and those
with $3$-prismatic symmetry form a two-parameter family. In each case we
identify all possible configurations of orbits of singular points in terms of
the parameters.

##  2 Preliminary results

We consider the vector space $\mathbb{S}^{n}$ of $n\times n$ complex symmetric
matrices. We denote by $W_{r}\subset\mathbb{P}(\mathbb{S}^{n})$ the projective
variety of matrices of rank at most $r$. Furthermore, by
$\mathbb{S}^{n}(\mathbb{R})\subset\mathbb{S}^{n}$ we denote the (real)
subspace of real symmetric matrices and $\mathbb{S}^{n}_{\succ 0}$,
$\mathbb{S}^{n}_{\succcurlyeq 0}\subset\mathbb{S}^{n}(\mathbb{R})$ stand for
the cones of positive definite and positive semidefinite matrices
respectively. Before we focus on the case $n=5$, we state some general
results. Let us first recall the following result from [6].

######  Lemma 2.1.

The codimension of $W_{r}$ in $\mathbb{P}(\mathbb{S}^{n})$ equals
$\binom{n-r+1}{2}$.

######  Definition 2.1.

Given a real point $p=[A]\in\mathbb{P}(\mathbb{S}^{\,n})$,
$A\in\mathbb{S}^{n}(\mathbb{R})$, the _signature_ of $p$ is the unordered pair
$\\{n_{+},n_{-}\\}$ where $n_{+}$ and $n_{-}$ are the numbers of positive
resp. negative eigenvalues of $A$. We call $p$ _spectrahedral_ if either
$n_{+}$ or $n_{-}$ is zero and _strictly spectrahedral_ if it has signature
$\\{n,0\\}$. For a (strictly) spectrahedral $p=[A]$ one can take
$A\in\mathbb{S}^{n}_{\succcurlyeq 0}$ (resp. $A\in\mathbb{S}^{n}_{\succ 0}$).

In the following, we assume $n\geq 3$. Let
$L\subset\mathbb{P}(\mathbb{S}^{n})$ be a sufficiently generic _real_ subspace
of dimension $4$, then $C=L\cap W_{n-2}$ is a curve.

######  Lemma 2.2.

The signature is constant on each connected component of $C({\mathbb{R}})$.

######  Proof 2.3.

Since $\operatorname{codim}(W_{n-3})=6$, every $p\in C$ has corank exactly
$2$. Therefore, because the eigenvalues of a symmetric matrix depend
continuously on their entries, we can find an open neighbourhood of each $p\in
C({\mathbb{R}})$ where the signature is constant. Thus the signature is
constant on each connected component.

######  Lemma 2.4.

Every connected component of $C({\mathbb{R}})$ which is spectrahedral either
realizes the trivial homology class in
$H_{1}(\mathbb{P}(\mathbb{S}^{n}(\mathbb{R}));{\mathbb{Z}}_{2})$ or is an
isolated point.

######  Proof 2.5.

The (real) hyperplane $H\subset\mathbb{P}(\mathbb{S}^{n})$ of all traceless
matrices contains no spectrahedral points. In particular, it is disjoint from
any spectrahedral connected component of $C({\mathbb{R}})$. This implies the
claim.

######  Corollary 2.

Let $V\subset\mathbb{P}(\mathbb{S}^{n})$ be a generic real linear subspace of
dimension $3$. Then $V\cap W_{n-2}$ consists of $\binom{n+1}{3}$ (complex)
points. The number of real points in $V\cap W_{n-2}$ that are spectrahedral is
even. Furthermore, the number of real points in $V\cap W_{n-2}$ is of the same
parity as ${n+1\choose 3}$.

######  Proof 2.6.

The degree of $W_{n-2}$ was computed in [6, Prop. 12(b)]. By genericity we can
write $V\cap W_{n-2}$ as the intersection of $C$ with a hyperplane that does
not contain any isolated real points of $C$. By Lemma 2.4 every spectrahedral
connected component of $C({\mathbb{R}})$ intersects a generic hyperplane in an
even number of points.

The last assertion follows from the fact that $V$ is real and nonreal points
in $V\cap W_{n-2}$ come in complex conjugate pairs.

######  Lemma 2.7.

For each $n\geq 4$ there is a full-dimensional set of real $3$-dimensional
linear subspaces $V\subset\mathbb{P}(\mathbb{S}^{n})$ which contain a strictly
spectrahedral point such that $V\cap W_{n-2}$ contains no spectrahedral
points.

######  Proof 2.8.

We proceed by induction on $n$. For $n=4$ this was shown in [8]. Assume that
$V\subset\mathbb{P}(\mathbb{S}^{n})$ is such a real $3$-dimensional linear
subspace. For $A\in\mathbb{S}^{n}$ let

$A^{\prime}=\begin{pmatrix}A&0\\\ 0&\operatorname{tr}(A)\end{pmatrix}.$

We claim that the subspace $V^{\prime}\subset\mathbb{P}(\mathbb{S}^{n+1})$ of
all $[A^{\prime}]$ with $[A]\in V$ has the desired properties. Clearly $[A]$
is (strictly) spectrahedral if and only if $[A^{\prime}]$ is (strictly)
spectrahedral. Thus $V^{\prime}$ contains a strictly spectrahedral point
because $V$ contains such a point. Now assume that there is a matrix $0\neq
A\in\mathbb{S}^{n}$ such that $[A^{\prime}]\in V^{\prime}\cap W_{n-1}$ is
spectrahedral. This implies that $A$ is positive (or negative) semidefinite
and thus $\operatorname{tr}(A)\neq 0$. Because $A^{\prime}$ has corank at
least $2$, the matrix $A$ has corank at least $2$ contradicting the assumption
that $V\cap W_{n-2}$ contains no spectrahedral points. Any small perturbation
of $V^{\prime}$ also has the desired properties.

Whereas for $n\geq 4$ there always exist generic real $3$-dimensional linear
subspaces $V\subset\mathbb{P}(\mathbb{S}^{n})$ such that $V\cap W_{n-2}$
contains no spectrahedral points, only for some special values of $n$ the real
locus of $V\cap W_{n-2}$ can be empty.

######  Lemma 2.9.

There exists a real $3$-dimensional linear subspace
$V\subset\mathbb{P}(\mathbb{S}^{n})$ containing a strictly spectrahedral point
and such that the real locus of $V\cap W_{n-2}$ is empty if and only if
$n=-1,0,1\ \mathrm{mod}\ 8$.

######  Proof 2.10.

Let $p=[C^{\mathsf{T}}C]\in V$ be a strictly spectrahedral point in $V$, where
$C$ denotes a nonsingular real $n\times n$ matrix. Since congruence by a non-
singular matrix leaves the varieties $W_{r}$ invariant, we have for the
_congruent space_

$V_{C}=\left\\{[(C^{\mathsf{T}})^{-1}AC^{-1}]\,:\,[A]\in V\right\\}$

that $V_{C}\cap W_{n-2}=(V\cap
W_{n-2})_{C}=\\{[(C^{\mathsf{T}})^{-1}AC^{-1}]\,:\,[A]\in V\cap W_{n-2}\\}$.
We can assume that $C=\mathrm{Id}$ is the identity matrix, otherwise we
replace $V$ with $V_{C}$ and $p=[C^{\mathsf{T}}C]\in V$ with $[\mathrm{Id}]\in
V_{C}$. Let $\mathrm{Id},A_{1},A_{2},A_{3}\in\mathbb{S}^{n}(\mathbb{R})$ be a
basis of the linear subspace $V\subset\mathbb{P}(\mathbb{S}^{n})$. Observe
that for $(t,\bar{x})=(t,x_{1},x_{2},x_{3})\in{\mathbb{R}}^{4}$, a real matrix

$A(t,\bar{x})=t\,\mathrm{Id}+x_{1}A_{1}+x_{2}A_{2}+x_{3}A_{3}$

is of corank at least two (that is, $[A(t,\bar{x})]\in V\cap W_{n-2}$) if and
only if the matrix

$A(0,\bar{x})=x_{1}A_{1}+x_{2}A_{2}+x_{3}A_{3}$

has a repeated (multiple) eigenvalue. Therefore, there exists a real linear
subspace $V\subset\mathbb{P}(\mathbb{S}^{n})$ of dimension $3$
($V=\left\\{[A(t,\bar{x})]\,:\,(t,\bar{x})=(x_{0},x_{1},x_{2},x_{3})\right\\}$
without loss of generality) as in the statement if and only there is a real
$2$-dimensional plane $v\subset\mathbb{P}(\mathbb{S}^{n})$
($v=\left\\{[A(0,\bar{x})]\,:\,\bar{x}=(x_{1},x_{2},x_{3})\right\\}$)
containing only real matrices with simple (pairwise distinct) eigenvalues. By
the main result from [5] existence of such a $2$-plane in
$\mathbb{P}(\mathbb{S}^{n})$ is equivalent to $\sigma(n)>2$, where

$\sigma(n)=\begin{cases}2,\ &n\neq 0,\pm 1\ (\mathrm{{mod}}\ 8),\\\ \rho(4b),\
&n=8b,8b\pm 1.\end{cases}$ (2)

Here $\rho(m)$ is the _Radon-Hurwitz number_ of the natural number $m$, that
is,

$\begin{aligned} \rho(m)=2^{c}+8d,\quad m=(2a+1)2^{c+4d}\end{aligned},$ (3)

where $a,c,$ and $d$ are integers and $c\in\\{0,1,2,3\\}$. Finally, for
$n=8b,8b\pm 1$ one always has $\sigma(n)=\rho(4b)>2$, which is straightforward
to check using (3).

In general, the real part of the projective hypersurface defined by a
_hyperbolic polynomial_ of degree $n$ can have at most
$\lfloor\frac{n+1}{2}\rfloor$ connected components with equality only if the
hypersurface has no real singular points.

######  Corollary 3.

Let $V\subset\mathbb{P}(\mathbb{S}^{n})$ be a real $3$-dimensional linear
subspace containing a strictly spectrahedral point. Then the real locus of
$V\cap W_{n-1}$ contains at most $\lfloor\frac{n+1}{2}\rfloor$ connected
components. Moreover, this bound is attained for some
$V\subset\mathbb{P}(\mathbb{S}^{n})$ if $n=-1,0,1$ mod $8$.

For small $n$ we describe how many connected components the real locus of
$V\cap W_{n-1}$ can have.

######  Corollary 4.

Let $V\subset\mathbb{P}(\mathbb{S}^{n})$ be as in Corollary 3. Then for $n<4$
the real locus of $V\cap W_{n-1}$ is connected, while for $n=4,5$ it is either
connected or has two connected components.

######  Proof 2.11.

The cases of $n=2,3,4$ follow from the classification in [8]. For $n=5$
Corollary 3 gives an upper bound of three components. However, the zero set of
a hyperbolic polynomial of degree $5$ can have three connected components only
if its real part is smooth and Lemma 2.9 shows that this is never the case for
$V\subset\mathbb{P}(\mathbb{S}^{5})$. It follows from Theorem 1 that there
exist subspaces in $\mathbb{P}(\mathbb{S}^{5})$ of types $(\rho,\sigma)=(2,2)$
and $(\rho,\sigma)=(4,2)$. In the first case $(V\cap W_{4})({\mathbb{R}})$ has
two connected components, while in the second case it is connected. See Figure
3 that illustrates the two possible cases.

######  Remark 5.

Let $L\subset\mathbb{P}(\mathbb{S}^{n}(\mathbb{R}))$ be a one-dimensional real
linear space which contains a strictly spectrahedral point. Then $L$ will
contain $n$ points of rank $n-1$ (counted with multiplicity). Since for real
symmetric matrices geometric multiplicities equal algebraic multiplicities,
any intersection point of $L$ with $W_{n-1}$ of multiplicity $>1$ will already
be a point in $W_{n-2}$.

Naively, one might expect that the same happens for real linear subspaces
$V\subset\mathbb{P}(\mathbb{S}^{n}(\mathbb{R}))$ of dimension $3$ containing a
striclty spectrahedral point. Namely, that any intersection point of such $V$
with $W_{n-2}$ of multiplicity $>1$ has rank at most $n-3$. But this is not
the case. For example let $V$ be the linear subspace spanned by the four
matrices:

$\left(\begin{smallmatrix}2&0&0&0&0\\\ 0&2&0&0&0\\\ 0&0&1&0&0\\\ 0&0&0&0&0\\\
0&0&0&0&0\end{smallmatrix}\right),\left(\begin{smallmatrix}0&0&0&0&0\\\
0&0&0&0&0\\\ 0&0&2&1&0\\\ 0&0&1&2&0\\\
0&0&0&0&2\end{smallmatrix}\right),\left(\begin{smallmatrix}1&0&-1&1&0\\\
0&1&0&0&1\\\ -1&0&2&-1&0\\\ 1&0&-1&1&0\\\
0&1&0&0&1\end{smallmatrix}\right),\left(\begin{smallmatrix}1&0&1&0&1\\\
0&1&0&-1&0\\\ 1&0&3&0&1\\\ 0&-1&0&1&0\\\ 1&0&1&0&1\end{smallmatrix}\right).$

One checks that $V$ intersects $W_{3}$ in $5$ different points, all of them of
multiplicity $4$, spectrahedral, and of rank exactly $3$. The associated
symmetroid is the union of a cubic and a quadratic hypersurface which
intersect in a sextic curve. This sextic curve is the union of two (nonreal)
complex conjugate rational normal curves (of degree $3$). The $5$ points of
rank $3$ are exactly the points where these two rational normal curves
intersect.

Let $V\subset\mathbb{P}(\mathbb{S}^{n})$ be a generic real subspace of
dimension $3$ containing a strictly spectrahedral point. Every spectrahedral
symmetroid is of the form $V\cap W_{n-1}$. Recall that by $\rho$ and $\sigma$
we denote the number of real and spectrahedral nodes on the symmetroid $V\cap
W_{n-1}$. The pair $(\rho,\sigma)$ is called the _combinatorial type_ of the
symmetroid $V\cap W_{n-1}$ or, simply, of $V$. Summarizing some of the above
results we give necessary conditions on possible combinatorial types.

######  Corollary 6.

The combinatorial type $(\rho,\sigma)$ of a transversal spectrahedral
symmetroid of degree $n$ satisfies the following conditions:

  1. i)

$0\leq\sigma\leq\rho\leq{n+1\choose 3}$,

  2. ii)

$\sigma$ is even,

  3. iii)

$\rho$ and ${n+1\choose 3}$ are of the same parity,

  4. iv)

$\rho$ can be zero if and only if $n=-1,0,1\ \text{mod}\ 8$.

In [3] and [8] it was shown that for $n=4$ these are the only restrictions.
Theorem 1 asserts that also for the quintic ($n=5$) case the above conditions
on combinatorial types are sufficient. In order to prove sufficiency we
provide explicit examples of symmetroids realizing each combinatorial type. We
first describe in Section 3 how we found some of the types by making use of
symmetries. Realization of the remaing types is discussed in detail in Section
4.

##  3 Quintic spectrahedra with symmetry

Let $G$ be a finite group. Let $V$ and $U$ be a $4$ resp. $5$-dimensional
represention of $G$. If we have a $G$-linear map
$\varphi:V\to\operatorname{Sym}_{2}(U)$, then the preimage under $\varphi$ of
the set of positive semidefinite symmetric bilinear forms on $U$ is a
spectrahedron invariant under the action of $G$. The same holds true for the
preimage of the corank $k$ locus for any $k=0,\ldots,5$.

######  Lemma 3.1.

If there is a strictly spectrahedral point, then the invariant part $V^{G}$ of
$V$ contains a strictly spectrahedral point. In particular, $V^{G}$ is
nontrivial.

######  Proof 3.2.

Let $v\in V$ be a strictly spectrahedral point, i.e. $\varphi(v)$ is positive
definite. Since the action of $G$ on $\operatorname{Sym}_{2}(U)$ preserves
positive definiteness, the symmetric bilinear form

$\varphi\left(\sum_{g\in G}g.v\right)=\sum_{g\in G}g.\varphi(v)$

is positive definite as well. Now the claim follows from $\sum_{g\in G}g.v\in
V^{G}$.

This shows that $V$ must contain (at least) one copy of the trivial
represention of $G$. The remaining $3$-dimensional space $V^{\prime}$
corresponds to the symmetries of a $3$-polytope. One can check that neither
octahedral symmetry nor icosahedral symmetry give rise to transversal quintic
symmetroids.

###  3.1 Tetrahedral symmetry

The symmetry group of the tetrahedron is the symmetric group
${\mathfrak{S}}_{4}$ and the corresponding representation $V^{\prime}$ is the
standard representation. Thus $V$ is just given by ${\mathfrak{S}}_{4}$
permuting the coordinates of ${\mathbb{R}}^{4}$. If we denote by $V_{\lambda}$
the irreducible representation of ${\mathfrak{S}}_{4}$ corresponding to the
partition $\lambda$, then ${\mathbb{R}}^{4}=V_{4}\oplus V_{3,1}$. In this
case, the only possibility to obtain transversal symmetroids is when
$U=V_{3,1}\oplus V_{2,2}$. After applying some suitable changes of bases on
both sides, we can then assume that the images of the unit vectors in
${\mathbb{R}}^{4}$ under $\varphi$ are given by the four matrices

$\left(\begin{smallmatrix}16+2t&24+t&24+t&-8&-8\\\ 24+t&16+2t&24+t&8&0\\\
24+t&24+t&16+2t&0&8\\\ -8&8&0&16&8\\\
-8&0&8&8&16\end{smallmatrix}\right),\left(\begin{smallmatrix}16+2t&-8+t&-8+t&8&8\\\
-8+t&-16+2t&-8+t&8&16\\\ -8+t&-8+t&-16+2t&16&8\\\ 8&8&16&16&8\\\
8&16&8&8&16\end{smallmatrix}\right),$
$\left(\begin{smallmatrix}-16+2t&-8+t&-8+t&-8&8\\\ -8+t&16+2t&-8+t&-8&0\\\
-8+t&-8+t&-16+2t&-16&-8\\\ -8&-8&-16&16&8\\\
8&0&-8&8&16\end{smallmatrix}\right),\left(\begin{smallmatrix}-16+2t&-8+t&-8+t&8&-8\\\
-8+t&-16+2t&-8+t&-8&-16\\\ -8+t&-8+t&16+2t&0&-8\\\ 8&-8&0&16&8\\\
-8&-16&-8&8&16\end{smallmatrix}\right)$

for some real number $t\in{\mathbb{R}}$. For generic values of $t$ there are
$20$ (complex) points in the rank $3$ locus. In this case, the $20$ points
split up into three ${\mathfrak{S}}_{4}$-orbits, one of length $12$ and two of
length $4$, and we have the following cases. If $48<t$, then the two
$4$-orbits consist of semidefinite real points. The $12$-orbit consists of
indefinite real points (Figure 1, yellow). If $12<t<48$, then one $4$-orbit
consists of semidefinite real points and the other $4$-orbits of indefinite
real points. The $12$-orbit consists of semidefinite real points (Figure 1,
green). If $0<t<12$, then one $4$-orbit consists of semidefinite real points
and the other $4$-orbit of indefinite real points. The $12$-orbit consists of
nonreal points (Figure 1, red). If $-2<t<0$, then the two $4$-orbits consist
of indefinite real points. The $12$-orbit consists of nonreal points. The
spectrahedron is empty (Figure 1, blue). If $t<-2$, then none of the $20$
points are real. The spectrahedron is empty (Figure 1, purple).

Figure 1: Quintic symmetroids with tetrahedral symmetry.

###  3.2 3-prismatic symmetry

The symmetry group of the triangular prism is
$G={\mathfrak{S}}_{3}\times{\mathfrak{S}}_{2}$ where ${\mathfrak{S}}_{3}$ is
the symmetry group of the triangle and ${\mathfrak{S}}_{2}$ corresponds to a
reflection that interchanges the two triangles of the prism. Again one
calculates that (after applying some suitable changes of bases on both sides)
the images of the unit vectors in ${\mathbb{R}}^{4}$ under $\varphi$ are given
by the four matrices

$\left(\begin{smallmatrix}1&a&a&0&0\\\ a&b+10&b+5&0&0\\\ a&b+5&b+10&0&0\\\
0&0&0&3&2\\\
0&0&0&2&3\end{smallmatrix}\right),\left(\begin{smallmatrix}1&-a&0&0&0\\\
-a&b+10&5&0&0\\\ 0&5&10&0&0\\\ 0&0&0&3&1\\\
0&0&0&1&2\end{smallmatrix}\right),\left(\begin{smallmatrix}1&0&-a&0&0\\\
0&10&5&0&0\\\ -a&5&b+10&0&0\\\ 0&0&0&2&1\\\
0&0&0&1&3\end{smallmatrix}\right),\left(\begin{smallmatrix}0&0&0&0&0\\\
0&0&0&2&1\\\ 0&0&0&1&2\\\ 0&2&1&0&0\\\ 0&1&2&0&0\end{smallmatrix}\right).$

For generic values of $a,b$ there are $20$ (complex) points in the rank $3$
locus. In this case, the $20$ points split up into five $G$-orbits, one of
length $2$, three of length $4$ and one of length $6$. The following table
lists all different types we can get in this way together with one value for
$(a,b)$ realizing this type:

Type | Spectrahedron | a | b | Region
---|---|---|---|---
| nonempty? |  |  |
(20,14) | Yes | $\frac{1}{2}$ | $-\frac{3}{2}$ | Brown
(20,8) | Yes | 2 | 6 | Blue
(18,6) | Yes | $\frac{1}{2}$ | 4 | Gray
(14,14) | Yes | $\frac{1}{2}$ | -2 | Orange
(14,8) | Yes | 1 | 1 | Yellow
(14,2) | Yes | 1 | 6 | Red
(10,4) | Yes | 1 | -2 | Cyan
(8,2) | Yes | $\frac{1}{2}$ | 1 | Green
(2,2) | Yes | $\frac{1}{3}$ | -2 | Purple
(6,0) | No | 2 | -16 | n.a.
(0,0) | No | 1 | -16 | n.a.
Figure 2: The discriminant (black) divides the $(a,b)$-plane into different
regions. The color of each region indicates the type of the symmetroids that
it contains.

Figure 3: Two quintic symmetroids with $14$ nodes on the spectrahedron. The
one on the left has $6$ additional real nodes. The real part of the one on the
right is disconnected.

##  4 Computations

We show that the conditions in Theorem 1 are sufficient by exhibiting explicit
examples of each possible combinatorial type. We explain how we used numerical
algebraic geometry to find these examples and certify their correctness. First
we describe equations which cut out the nodes of quintic symmetroids.

###  4.1 Equations

We consider full-dimensional spectrahedra so without loss of generality, we
assume that our linear space of symmetric matrices
$V_{A}\subset\mathbb{S}^{5}$ is of the form

$A(t,\bar{x}):=t\mathrm{Id}+x_{1}A_{1}+x_{2}A_{2}+x_{3}A_{3}.$

These linear spaces are parametrized by triples
$A=(A_{1},A_{2},A_{3})\in(\mathbb{S}^{5})^{3}$ such that
$\mathrm{Id},A_{1},A_{2},A_{3}$ are linearly independent. The symmetroid
$V_{A}\cap W_{4}$ is the set of zeros of
$D_{A}(t,\bar{x}):=\det(A(t,\bar{x}))$ and generically the $20$ nodes
$V_{A}\cap W_{3}$ are the solutions to the polynomial system

$\frac{\partial}{\partial t}D_{A}(t,\bar{x})=\frac{\partial}{\partial
x_{1}}D_{A}(t,\bar{x})=\frac{\partial}{\partial
x_{2}}D_{A}(t,\bar{x})=\frac{\partial}{\partial x_{3}}D_{A}(t,\bar{x})=0.$ (4)

######  Remark 7.

Since the solutions to this system are points in $\mathbb{P}^{3}$, there are
always at least $20$ complex projective solutions to (4) counted with
multiplicity. This number may be larger (or infinite) at nongeneric values of
$A$, but never smaller.

Although the solutions to (4) are projective points, in practice we compute
them in a generic real affine chart represented by an equation of the form
$\ell(t,\bar{x})=c_{0}t+c_{1}x_{1}+c_{2}x_{2}+c_{3}x_{3}+c_{4}$, where
$c_{i}\in\mathbb{R}$. Let $F_{A}(t,\bar{x})$ denote the polynomial system (4)
together with $\ell(t,\bar{x})=0$. Each solution $s$ to $F_{A}(t,\bar{x})$
corresponds to a matrix $A(s)$. When
$A\in(\mathbb{S}^{5}(\mathbb{{\mathbb{R}}}))^{3}$, the following conditions
concerning reality coincide:

$s\in\mathbb{R}^{4}\iff
A(s)\in\mathbb{S}^{5}(\mathbb{R})\iff[A(s)]\cap\mathbb{P}(\mathbb{S}^{5}(\mathbb{R}))\neq\emptyset.$

One checks whether $A(s)$ is semidefinite either by checking that its
eigenvalues have the same sign, or by assessing the signs of its principal
minors.

###  4.2 Numerical algebraic geometry

We use the julia package HomotopyContinuation.jl [2] to solve instances of
$F_{A}(t,\bar{x})$ for various parameters $A$. The equations
$F_{A}(t,\bar{x})$ form a parametrized polynomial system which can be solved
using a _parameter homotopy_. This process involves numerically solving
$F_{A^{0}}(t,\bar{x})$ for random _complex_ symmetric matrices $A^{0}$.
Afterwards, the method of _homotopy continuation_ is used to solve any
instance of $F_{A}(t,\bar{x})$ with $A\in(\mathbb{S}^{5}(\mathbb{R}))^{3}$
quickly using a _parameter homotopy_ from the start system
$F_{A^{0}}(t,\bar{x})$ to the target system $F_{A}(t,\bar{x})$. The following
snippet of code solves $20000$ instances of $F_{A}(t,\bar{x})$ in about $30$
minutes.

using HomotopyContinuation, LinearAlgebra @var x[1:4], a[1:3,1:5,1:5]
A=vcat([LinearAlgebra.I],[Symmetric(a[i,:,:]) for i in 1:3]) Ax=sum([x[i]*A[i]
for i in 1:4]) affine_chart=sum(x.*randn(Float64,4))+randn()
eqs=vcat([differentiate(det(Ax),x[i]) for i in 1:4],affine_chart)
quintic_system=System(eqs;parameters=unique(vcat(vec(A[2]),vec(A[3]),vec(A[4]))))
init_params=randn(ComplexF64,45)
init_sols=solve(quintic_system,target_parameters=init_params)
samples=solve(quintic_system, solutions(init_sols),
start_parameters=init_params, target_parameters=[randn(Float64,45) for i in
1:20000]); Solving for 20000 parameters... 100%|||||||||||||||||| Time:
0:31:51 # parameters solved: 20000 # paths tracked: 400000

###  4.3 A hill-climbing algorithm

Finding the maximal number of real solutions in a parametrized polynomial
system is a popular problem in applications. One very successful method for
doing this is a hill-climbing algorithm developed by Dietmaier [4]. We adapt
his idea to our setting, where we care about more than the number of real
solutions to $F_{A}(t,\bar{x})$: we want to find all combinatorial types.

To best explain our method, we first describe what a hill-climbing algorithm
is. The input of a hill-climbing algorithm is a parameter space $\mathcal{P}$
equipped with a metric, and an objective function $\varphi:\mathcal{P}\to R$
where $R$ is some ordered set. The goal of the algorithm is to maximize
$\varphi$ over $\mathcal{P}$ and the steps are simple. Begin at some
$P_{*}\in\mathcal{P}$ and compute $\varphi_{*}=\varphi(P_{*})$. Compute the
values of $\varphi$ evaluated at many parameters near $P_{*}$ and update
$P_{*}$ to be the one which evaluates to the largest value when this is larger
than $\varphi_{*}$, then repeat. In this fashion, the algorithm greedily
travels through $\mathcal{P}$ increasing the value of $\varphi$.

In [4], $\mathcal{P}$ is the parameter-space of a real parametrized polynomial
system $G_{\mathcal{P}}=0$. The set $R$ is $\mathbb{N}\times{\mathbb{R}}$
equipped with the lexicographic ordering. The objective function takes a
parameter $P$ to $(\rho,-m)$ where $\rho$ is the number of real solutions to
$G_{P}=0$, and $m$ is the minimum norm of the complex part of a nonreal
solution of $G_{P}=0$. Consequently, this algorithm tries to greedily push
complex conjugate solutions together to make them real, while always prefering
parameters with more real solutions.

0022446688101012121414161618182020 Figure 4: All combinatorial types of
quintic spectrahedra.

To find each combinatorial type in Figure 4, we take a similar approach. After
solving $F_{A}(t,\bar{x})$ for some parameter $A$, we use four different hill-
climbing algorithms, each of which attempts to move the combinatorial type of
$V_{A}$ in one of the directions displayed in Figure 4.

We introduce some notation. For any two combinatorial types
$\tau=(\rho,\sigma)$ and $\tau^{\prime}=(\rho^{\prime},\sigma^{\prime})$, let
$d(\tau,\tau^{\prime})$ be their lattice distance in the lattice generated by
the vectors in Figure 4. Given a nonreal matrix
$B=B_{\mathbb{R}}+\sqrt{-1}B_{\mathbb{C}}$, let
$\mu(B)=||B_{\mathbb{C}}||_{2}$. Let $\eta(B)=1$ if the largest three
eigenvalues of $B_{\mathbb{R}}$ have the same sign and $0$ otherwise. For
$A\in(\mathbb{S}^{5}(\mathbb{R}))^{3}$ let $S_{{\mathbb{R}}_{+}}(A)$ and
$S_{{\mathbb{R}}_{-}}(A)$ denote the real semidefinite and real indefnite
solutions to $F_{A}(t,\bar{x})$. Similarly, let $S_{{\mathbb{C}}_{+}}(A)$
denote the collection of nonreal solutions $B$ with $\eta(B)=1$ and
$S_{{\mathbb{C}}_{-}}(A)$ the nonreal solutions with $\eta(B)=0$. For every
$A\in(\mathbb{S}^{5}(\mathbb{R}))^{3}$, we have the functions

$\displaystyle\mu_{+}(A)$ $\displaystyle=\min_{B\in
S_{{\mathbb{C}}_{+}}(A)}(\mu(B))\hskip
36.135pt\delta_{+}(A)=\min_{B_{1},B_{2}\in
S_{{\mathbb{R}}_{+}}(A)}(||B_{1}-B_{2}||_{2})$ $\displaystyle\mu_{-}(A)$
$\displaystyle=\min_{B\in S_{{\mathbb{C}}_{-}}(A)}(\mu(B))\hskip
36.135pt\delta_{-}(A)=\min_{B_{1},B_{2}\in
S_{{\mathbb{R}}_{-}}(A)}(||B_{1}-B_{2}||_{2})$

which formally evaluate to $+\infty$ when the minimum is taken over the empty
set.

With these definitions, the following table describes the four hill-climbing
algorithms used to move from some type $\tau$ to another type
$\tau^{\prime}=\tau+$Direction.

Direction | $R$ | $\varphi$ | Ordering
---|---|---|---
$(+2,+2)$ | $\mathbb{N}\times\mathbb{R}\cup\\{\infty\\}$ | $(d(\tau^{\prime},(\rho,\sigma)),-\mu_{+}(A))$ | Lex
$(+2,0)$ | $\mathbb{N}\times\mathbb{R}\cup\\{\infty\\}$ | $(d(\tau^{\prime},(\rho,\sigma)),-\mu_{-}(A))$ | Lex
$(-2,-2)$ | $\mathbb{N}\times\mathbb{R}\cup\\{\infty\\}$ | $(d(\tau^{\prime},(\rho,\sigma)),-\delta_{+}(A))$ | Lex
$(-2,0)$ | $\mathbb{N}\times\mathbb{R}\cup\\{\infty\\}$ | $(d(\tau^{\prime},(\rho,\sigma)),-\delta_{-}(A))$ | Lex

Rows correspond to arrows in Figure 4 and describe algorithms which attempt to
push two solutions (in $S_{{\mathbb{C}}_{+}}$,$S_{{\mathbb{C}}_{-}}$,
$S_{{\mathbb{R}}_{+}}$, or $S_{{\mathbb{R}}_{-}}$ respectively) together.

Since hill-climbing methods get stuck at local minima, it was often necessary
to restart the algorithm with new initial values many times until one became
successful. For some of the most difficult cases like those near $(20,20)$ and
$(20,0)$, monitoring of the algorithm was required due to ill-conditioning.

###  4.4 Certification

Solving $F_{A}(t,\bar{x})$ numerically only provides approximate solutions and
so _a posteriori_ certification methods are necessary in order to verify how
many of these solutions are real, how many correspond to semidefinite
matrices, and ultimately whether the solutions are even correct.

Certifying the correctness and reality of a set of solutions to a polynomial
system is usually simple and can be performed via the command certify [1] in
HomotopyContinuation.jl. The input of this command is a collection $S$ of
approximate solutions to a polynomial system $F$, and when successful, the
output is a collection of $|S|$ bounding boxes, each containing a unique
solution to $F$. This method, however, cannot be immediately applied to our
situation because the system $F_{A}(t,\bar{x})$ is not _square_ : there are
more equations than variables. Nor does this method verify which solutions
correspond to semidefinite matrices.

To remedy these issues, we solve a different polynomial system
$G_{A}(t,\bar{x},d,M)$:

$\displaystyle\frac{\partial}{\partial x_{1}}D_{A}(t,\bar{x})$
$\displaystyle=0\hskip 23.84923pt\frac{\partial}{\partial
x_{2}}D_{A}(t,\bar{x})=0\hskip 12.64746pt\frac{\partial}{\partial
x_{3}}D_{A}(t,\bar{x})=0$ (5) $\displaystyle\ell(t,\bar{x})$
$\displaystyle=0,\hskip 23.12692pt\det(A(t,\bar{x}))=d,\hskip
14.45377pt(A(t,\bar{x}))_{I}=M_{I}.$

The system $G_{A}(t,\bar{x},d,M)$ consists of three out of four of the
equations of (4), the affine chart $\ell(t,\bar{x})$, an equation tracking the
value of the determinant of $A(t,\bar{x})$, and equations tracking each of the
values of the $5+10+10=25$ principal minors $M_{I}$ indexed by subsets $I$ of
$\\{1,\ldots,5\\}$ of sizes $1$, $2$, and $3$. In total,
$G_{A}(t,\bar{x},d,M)$ is a polynomial system consisting of $30$ equations in
$30$ unknowns. Since it is square, we can certify its solutions.

The total number of solutions to the first three equations is $64$. This is
their Bézout bound. The other equations and variables form a graph of a
function over this set. When distinct, at least $20$ of these $64$ solutions
project to solutions of $F_{A}(t,\bar{x})$ by Remark 7. Those which do are the
points whose $d$-coordinate equals zero. Therefore, showing that $44$ of the
solutions do not have $0$ in their certified bounding interval in the
$d$-coordinate proves that the other $20$ solutions have determinant equal to
zero. Similarly, one certifies the signs of the minors whenever $0$ is not
contained in the corresponding bounding intervals. These facts suggest an
algorithm for certifying combinatorial.

######  Algorithm 4.1.

.
Input: Three real symmetric matrices $A=(A_{1},A_{2},A_{3}).$
Output: The combinatorial type $(\rho,\sigma)$ of $V_{A}$, or Unsuccessful.

  1. 1.

Solve $G_{A}(t,\bar{x},d,M)$ numerically to obtain $S$, consisting of $64$
solutions.

  2. 2.

Run certify to obtain $64$ bounding boxes for these solutions.

  3. 3.

Delete $44$ bounding boxes which do not contain $0$ in the $d$-coordinate.

  4. 4.

Count $\rho$ to be the number of remaining boxes which are real.

  5. 5.

For each real box, determine the signs of the $M$-coordinates.

  6. 6.

Count $\sigma$ to be the number of boxes whose $M$-coordinates imply
semidefiniteness of the solution.

  7. 7.

If successful, return $(\rho,\sigma)$, otherwise, return Unsuccessful.

We successfully performed Algorithm 4.1 on witnesses for each of the $65$
combinatorial types. Certificates and necessary software for reproducibility
may be found at

$\href https://mathrepo.mis.mpg.de/QuinticSpectrahedra/index.html$ (6)

Additionally, (6) contains a gallery of images displaying examples of the $65$
combinatorial types. Figure 5 shows two extremal cases: $(20,20)$ and
$(20,0)$.

Figure 5: Quintic spectrahedra of types $(20,20)$ and $(20,0)$.

## Acknowledgements

This work is a part of the collaboration project “Linear Spaces of Symmetric
Matrices” at MPI MiS and worldwide. We would like to thank Orlando Marigliano,
Mateusz Michałek, Kristian Ranestad, Tim Seynnaeve, and Bernd Sturmfels for
coordinating the project and inspiring the present work. We would also like
thank Cynthia Vinzant for her help in visualizing spectrahedra, as well as
Sascha Timme and Paul Breiding for their help with the certification.

## References

  * [1] P. Breiding, K. Rose and S. Timme: Certifying roots of polynomial systems using interval arithmetic,  arXiv preprint arXiv:2011.05000, 2020.
  * [2] P. Breiding and S. Timme, HomotopyContinuation.jl: A package for homotopy continuation in Julia, pp. 458–465 in Mathematical software — ICMS 2018 (South Bend, IN, 2018), edited by J.H. Davenport et al., Lecture Notes in Computer Science 10931, Springer 2018.
  * [3] A. Degtyarev and I. Itenberg, On real determinantal quartics, Proceedings Gökova Geometry-Topology Conference 2010, Int. Press, Somerville, MA (2011), 110–128.
  * [4] P. Dietmaier: The Stewart-Gough platform of general geometry can have 40 real postures, Lenarčič J., Husty M.L. (eds): Advances in Robot Kinematics: Analysis and Control, 7–16, Springer, Dordrecht, 1998.
  * [5] S. Friedland, J. W. Robbin and J. H. Sylvester, On the crossing rule, Communications on Pure and Applied Mathematics, 37 (1) (1984), 19–37.
  * [6] J. Harris and L. W. Tu, On symmetric and skew-symmetric determinantal varieties, Topology, 23 (1) (1984), 71–84.
  * [7] J. W. Helton and V. Vinnikov, Linear Matrix Inequality Representation of Sets, Communications on Pure and Applied Mathematics, 60 (5) (2007), 654–674.
  * [8] J. C. Ottem, K. Ranestad, B. Sturmfels and C. Vinzant, Quartic spectrahedra, Math. Program., 151 (2, Ser. B) (2015), 585–612.