aainstitutetext: University of Campinas – UNICAMP,
Campinas, 13083-859, Brazil

# Open-Source Numerical Solver for Neutrino Collective Effects - I: Isotropic
Neutrino Gas

P. Dedin Neto,111Corresponding author<EMAIL_ADDRESS>
###### Abstract

In this paper, we introduce a new open-source code to find numerical solutions
for the neutrino evolution considering neutrino-neutrino interactions, which
result in the so-called collective effects. We first describe the theoretical
background of this type of evolution, explaining the polarization vector
formalism in which we implement our numerical code. We then show the results
for different neutrino systems. In this first paper, we focus on the case of
an isotropic neutrino gas, exploring a mono-energetic scenario and one with a
spectral distribution. The resulting code of this work is available at
https://github.com/pedrodedin/Neutrino-Collective-Effects.

##  1 Introduction

When considering environments with a high density of neutrinos, such as the
case of a supernova and the early universe, we have to consider the potential
due to neutrino-neutrino forward scattering. However, the inclusion of
neutrino-neutrino interactions makes the evolution equation nonlinear and
complicated to solve in many cases. For supernova neutrinos, it became
extremely challenging due to many complications (such as non-uniform emission,
energy and flavor-dependent neutrinospheres, among others) and is still an
unsolved problem. Despite the difficulty, there are many works in the
literature trying to solve this problem. Some of these works show analytical
solutions for simpler neutrino systems (Hannestad2006-zw, ; Duan:2005cp, ),
such as mono-energetic and isotropic neutrino gas. Others take the numerical
approach and consider more complex systems (Duan2006-dp, ) 222See
(Duan2010-dq, ; Mirizzi:2015eza, ) for reviews on the topic.. Even so, we
still do not have a clear picture of the neutrino evolution, given that new
phenomena emerge when other features are considered, such as the case of fast
oscillations due to ELN (electron lepton number) crossing in the angular
distribution (Tamborra:2020cul, ; Sawyer:2008zs, ).

In this context, this paper aims to describe a new open-source code for
numerical solutions to the neutrino evolution when considering neutrino-
neutrino interactions. In this first version, we explore the case of an
isotropic neutrino gas. This system is relatively simple and has already been
explored in the literature. However, when talking about collective effects,
there is a lack of open-source codes from the papers that present numerical
solutions, which makes it difficult to reproduce and analyze the results.
Therefore, our work is an effort to try to change this picture, offering a
code with which anyone can reproduce our results and make further
improvements. Moreover, we intend to continue to improve this code, exploring
increasingly supernova-like scenarios in future works.

This document is organized as follows. In section 2, we describe the
theoretical background of the neutrino evolution equation, in which we
consider a free-streaming neutrino influenced by forward scattering potentials
due to neutrino-electron and neutrino-neutrino interactions. In the same
section, we describe the framework of polarization vectors that we use both in
the code implementation and to analyze the results. In section 3, we describe
how the numerical implementation is done, listing the important steps of the
code and the libraries used. In section 4 we show the numerical results for
the different scenarios considered up to this moment. Finally, in section 5 we
make our conclusions.

##  2 The Evolution Equation

###  2.1 Evolution Equation - Free-streaming Neutrino

For any quantum system, its evolution is given by the Hamiltonian. In the
Schrodinger picture, the evolution of a quantum state $\ket{\psi}$ is given by
the following equation:

$i\frac{d}{dt}\ket{\psi(t)}=H\ket{\psi(t)}=(H_{0}+H_{int})\ket{\psi(t)},$ (1)

where the Hamiltonian can be divided into a free $H_{0}$ and an interacting
part $H_{int}$ 333It is worth mentioning that we consider the time $t$ and
propagate distance $r$ as interchangeable throughout the paper, given that the
neutrino is ultra-relativistic ($E\gg m_{\nu}$) in the cases interesting for
us.. For a free-streaming neutrino, i.e., a neutrino that is not scattered
away from its trajectory from the source to the detector, the interaction part
of the Hamiltonian is composed of forward-scattering potentials. That is the
case of solar neutrinos and supernova neutrinos free-streaming above the
neutrinosphere. For supernova neutrinos, the relevant potentials are the ones
due to scattering in ordinary matter (electrons, protons, and neutrons) and
neutrinos themselves. For interactions with ordinary matter, if we consider
only active neutrinos, the potential resulting from scattering in protons and
neutrons is irrelevant, as it only contributes to an overall phase. In this
case, the only contribution that matters is the one due to the neutrino-
electron charged current (CC) interaction (Figure 1a), for which the potential
in the flavor basis is given by

$H_{\nu e}=\sqrt{2}G_{F}n_{e}\text{diag}\left[1,0,0\right]$ (2)

considering three families of neutrinos, where $G_{F}$ is the Fermi constant
and $n_{e}$ is the local electron density (wolfenstein1978neutrino, ). For the
neutrino-neutrino interactions (Figure 1b), the potential is given by the
following expression (Neto:2021hhl, ; sigl1993general, ):

$H_{\nu\nu}=\sum_{j=1}^{N}\sqrt{2}G_{F}(1-\cos\theta_{ij})[\rho_{j}(t)-\overline{\rho}_{j}(t)],$
(3)

where $\theta_{ij}$ is the angle between the momentum $\vec{p}_{i}$ of the
evolved neutrino and the momentum $\vec{p}_{j}$ of the $j$-th neutrino or
antineutrino interacting with the evolved one, such that their density
matrices are given respectively by
$\rho_{j}(t)=\ket{\nu_{j}(t)}\bra{\nu_{j}(t)}$ and
$\overline{\rho}_{j}(t)=\ket{\overline{\nu}_{j}(t)}\bra{\overline{\nu}_{j}(t)}$.
The indices $i$ and $j$ may as well be exchanged by a momentum $\vec{p}$
index, this just depends on which way we want to describe our neutrinos
(individually or by an ensemble of a given momentum). It is also worth
mentioning that the electrons and neutrinos interacting with $\ket{\psi(t)}$
are the ones that have a wave-packet overlap with it, so that the quantities
in the equations above, such as electron and neutrino densities, correspond to
their local value in the neutrino trajectory 444In this sense, it is useful to
imagine the neutrino wave function spatial distribution $\psi(\vec{r},t)$ when
looking to the equation (1), rather than just an abstract state $\ket{\psi}$.
A good discussion about the microscopic nature of the neutrino evolution can
be found on (Akhmedov:2020vua, ).

[vertical=a to b] i1 [particle=$\nu_{e}(p_{1})$] – [fermion] a – [fermion] f1
[particle=$e^{-}(p_{2})$], a – [boson, edge label=$W^{+}$] b, b – [fermion] i2
[particle=$\nu_{e}(p_{1})$], f2 [particle=$e^{-}(p_{2})$] – [fermion] b, ;

(a)

[vertical=a to b] i1 [particle=$\nu_{\alpha}(p_{1})$] – [fermion] a –
[fermion] f1 [particle=$\nu_{\alpha}(p_{1})$], a – [boson, edge label=$Z^{0}$]
b, b – [fermion] i2 [particle=$\nu_{\beta}(p_{2})$], f2
[particle=$\nu_{\beta}(p_{2})$] – [fermion] b, ;

(b)

Figure 1: Feynman diagrams for the neutrino forward scattering in electrons
and neutrinos.

###  2.2 Polarization Vector Formalism

Given the evolution equation (1), we will now translate it into the
polarization vector formalism, the one in which we implement our code. First,
we can describe a quantum state $\ket{\psi(t)}$ using the density matrix,
which is defined as follows:

$\rho(t)=\ket{\psi(t)}\bra{\psi(t)}.$ (4)

We can also define the density matrix for a statistical ensemble of states
(incoherent mixture), where each state $\ket{\psi_{i}(t)}$ have a statistical
weight $p_{i}$. The last case represents what is called a mixed state, and the
former case is called a pure state. Using equation (1), we can write the
evolution equation for the density matrix as

$i\frac{d\rho}{dt}=[H,\rho]$ (5)

If we consider a scenario of two families of neutrinos, we can decompose the
matrices above in Pauli matrices. Then, we can use the coefficients of this
expansion as components of three-dimensional vectors that represent the
matrices in the following way:

$H=-\frac{1}{2}\vec{\sigma}\cdot\vec{H}\text{,}\;\;\;\;\;\rho=\frac{1}{2}\mathbf{1}+\frac{1}{2}\vec{\sigma}\cdot\vec{P}\text{,}\;\;\;\;\;\vec{\sigma}=(\sigma_{1},\sigma_{2},\sigma_{3}).$
(6)

Here we ignore the Hamiltonian component proportional to the identity, which
is irrelevant for the evolution. Using this vector notation, the evolution
equation becomes a precession one

$\frac{d}{dt}\vec{P}(t)=\vec{P}(t)\times\vec{H}\;\;\;\leftrightarrow\;\;\;i\frac{d}{dt}P_{i}(t)=\frac{i}{2}P_{i}(t)H_{j}[\sigma_{i},\sigma_{j}].$
(7)

That is the differential equation that we implement in our code. Note that the
coefficients of expansion in the Pauli matrices $\sigma_{i}$ depend on the
basis that we are working on, where the 3rd component $P_{3}$ will give us the
content of eigenstates in the state $\ket{\psi(t)}$ for a given basis555The
Pauli matrix $\sigma_{3}$ is the only one with non-zero diagonal components..
Therefore, we just need to project $\vec{P}$ in the third component to get the
probability of detecting a specific eigenstate. The main motivations to work
in this formalism, as we will see in the results, are the nice pictorial
visualization of the neutrino evolution and the number of classical and well-
known equivalent systems.

###  2.3 Polarization Vectors and Forward-scattering Interactions ($H_{\nu e}$
and $H_{\nu\nu}$)

Let us now consider the specific case of free-streaming neutrinos where the
non-interacting component of the Hamiltonian is given by the vacuum
Hamiltonian $H_{V}$ and the interacting part is given by the potentials due to
forward scattering in electrons $H_{\nu e}$ and in neutrinos themselves
$H_{\nu\nu}$. In this case, we can write the evolution equation for a given
neutrino (labeled by the index $i$) as

$\begin{split}i\frac{d}{dt}\rho_{i}=[\omega H_{V}+\lambda H_{\nu e}+\mu
H_{\nu\nu,i},\rho_{i}],\\\ \omega\equiv\frac{\Delta
m^{2}}{2E_{i}},\;\;\;\;\lambda\equiv\sqrt{2}G_{F}n_{e},\;\;\;\;\mu\equiv\sqrt{2}G_{F}n_{\nu},\end{split}$
(8)

where $\omega$, $\lambda$, and $\mu$ give the characteristic intensity of its
respective component in the Hamiltonian. Again, we can translate the matrices
into vectors by decomposing them in Pauli matrices such that

$H_{V}=-\frac{1}{2}\vec{\sigma}\cdot\vec{B}\text{,}\;\;\;\;\;H_{\nu
e}=-\frac{1}{2}\vec{\sigma}\cdot\vec{L}\text{,}\;\;\;\;\;\rho=\frac{1}{2}\mathbf{1}+\frac{1}{2}\vec{\sigma}\cdot\vec{P}\text{.}$
(9)

With this decomposition, the evolution of the i-th neutrino and antineutrino
polarization vectors ($\vec{P}_{\nu_{i}}$ and $\vec{P}_{\overline{\nu}_{i}}$)
interacting with j (anti)neutrinos becomes

$\begin{split}\frac{d}{dt}\vec{P}_{\nu_{i}}=\vec{P}_{\nu_{i}}\times\left[\omega\vec{B}+\lambda\vec{L}-\mu\sum_{j}(1-\cos{\theta_{ij}})(\frac{n_{\nu,j}}{n_{\nu}}\vec{P}_{\nu,j}-\frac{n_{\overline{\nu},j}}{n_{\nu}}\vec{P}_{\overline{\nu},j})\right],\\\
\frac{d}{dt}\vec{P}_{\overline{\nu}_{i}}=\vec{P}_{\overline{\nu}_{i}}\times\left[-\omega\vec{B}+\lambda\vec{L}-\mu\sum_{j}(1-\cos{\theta_{ij}})(\frac{n_{\nu,j}}{n_{\nu}}\vec{P}_{\nu,j}-\frac{n_{\overline{\nu},j}}{n_{\nu}}\vec{P}_{\overline{\nu},j})\right].\end{split}$
(10)

Note that it is possible to redefine $\vec{P}$ as to be normalized by
$n_{\nu}$. One just needs to be careful when translating it back to the
density matrix or a conversion or survival probability. We can also turn this
equation more compact using the neutrino flavor isospin (NFIS) notation
$\vec{s}$ (Duan:2005cp, ). Even so, we will not use it in this work, keeping
the notation of polarization vectors evolved by equation (10). In the next
section, we describe how we implement this equation in our numerical code.

##  3 Numerical Implementation

Our code is implemented in Python and can be found in its respective
repositories on GitHub (https://github.com/pedrodedin/Neutrino-Collective-
Effects) and Zenodo (pedro_dedin_neto_2022_5964385, ), along with instructions
on how to use it. For the numerical solver, we use the scipy.integrate library
(2020SciPy-NMeth, ) through its odeint routine, which solves a system of
first-order ODEs (Ordinary Differential Equations) 666It is possible to solve
an n-order ODE passing it as a system of n first-order ODEs.. To solve a
system of $n$ ODEs, the solver requires a Python function that computes the
derivative $dy_{i}/dt=f_{i}(y,t,\theta)$ of the $n$ variables $y_{i}(t)$. It
also required an $n$-dimensional array $y_{0}$ with the initial conditions and
a time array $t$ with the points in which $y_{i}(t)$ will be solved.

In our case, the number of differential equations depends on how many
variables $\vec{P}_{\nu_{i}}$ depend on, that is, on how many different
polarization vectors we divide the neutrinos. In the simplest case, we have
only one polarization vector for each flavor of neutrinos and antineutrinos,
resulting in four vectors to be evolved. Considering two families of
neutrinos, each vector has three components, totalizing $n=12$ differential
equations. In a more general case, we may write the number of equations as
$n=2\times n_{f}\times n_{dim}\times n_{others}$, where $n_{f}$ is the number
of neutrino flavors, $n_{dim}$ the dimension of the space where the
polarization vectors live777In fact, the dimension $n_{dim}$ of this space can
be easily computed from the number of flavors $n_{f}$, given that the complex
$n_{f}\times n_{f}$ matrices are decomposed in the generators of the
$SU(n_{f})$ group, apart from the identity., and $n_{others}$ the other
possible degrees in which $\vec{P}_{\nu_{i}}$ or
$\vec{P}_{\overline{\nu}_{i}}$ could depend. For example, in the case of
spectral distribution, $n_{others}$ represents the number of energy bins.

With respect to the time sampling step $\Delta t$ used in the code, it needs
to be at least two times smaller than the smallest oscillation period that
composes the time variation of $y_{i}(t)$, as requested by the Nyquist-Shannon
sampling theorem. Therefore, a first bound is to consider a $\Delta t$ smaller
than the smallest vacuum oscillation period $2\pi/\omega_{max}$, which gives
us a good reconstruction for the vacuum oscillatory behavior. The other
potentials, such as $\lambda$ and $\mu$, and their variation may also give
rise to phenomena with a characteristic time shorter than the vacuum ones. For
these effects, instead of using an explicit expression for $\Delta t$ we make
empirical adjusts, using a sampling bellow the scale of the dominant potential
($\omega$, $\lambda$, or $\mu$). It is worth mentioning that the sampling step
passed to the ODE solver is only to sample the output, while the integration
step used to solve the system of equations is adjusted internally by the
solver routine.

##  4 Numerical Solutions for Different Scenarios

###  4.1 0th Case - Solar Neutrinos (MSW Effect)

Before diving into the cases with neutrino-neutrino interactions, we will
explore the neutrino evolution for the well-known case of solar neutrinos. We
first show the analytical solution and then compare it to our numerical one.
That serves as a test for the ODE numerical solver and for the framework that
we are using.

####  4.1.1 Analytical solution

In the solar case, $\nu_{e}$ neutrinos are created inside the sun in a high
density environment, which generates a matter potential due to neutrino-
electron forward scattering. This potential changes the mixing parameters
between the flavor states $\ket{\nu_{\alpha}}$ and the mass eigenstates in
matter $\ket{\nu^{M}_{i}}$. For two neutrino families, we have in the
following effective mixing angle and mass squared difference for a neutrino
propagating in matter:

$\tan{2\theta_{M}}=\frac{\tan{2\theta}}{1-\frac{A_{CC}}{\Delta
m^{2}\cos{2\theta}}},\;\;\;\;\;\Delta m^{2}_{M}=\sqrt{(\Delta
m^{2}\cos{2\theta}-A_{CC})^{2}+(\Delta m^{2}\sin{2\theta})^{2}},$ (11)
$A_{CC}\equiv\sqrt{2}G_{F}n_{e}E,\;\;\;\;\;A^{R}_{CC}=\Delta
m^{2}\cos{2\theta},$ (12)

where $A^{R}_{CC}$ defines the resonance condition, in which $\Delta
m^{2}_{M}$ is minimum and the mixing is maximum ($\theta_{M}=\pi/4$). As the
neutrino escapes the Sun, the electron density $n_{e}(r)$ changes over its
trajectory, leading to a flavor conversion described by the MSW (Mikheyev-
Smirnov-Wolfenstein) effect (wolfenstein1978neutrino, ; mikheev1985resonance,
). In summary, the evolution can be adiabatic if the electron density varies
slowly enough for the neutrino to remain in the same instantaneous mass
eigenstate, or non-adiabatic if the variation is fast enough to cause a
transition between the instantaneous eigenstates near the resonance region.
For the solar neutrinos, we have an adiabatic evolution, so that a neutrino
created in a specific superposition of effective mass eigenstates will stay in
this same superposition as the electron density decreases to zero. This
initial superposition will depend on the potential $A^{(i)}_{CC}$ at the
neutrino creation such that for an electron neutrino, we have the following
initial state:

$\ket{\nu_{e}(0)}=\cos{2\theta_{M}^{(i)}}\ket{\nu_{1}^{M}}+\sin{2\theta_{M}^{(i)}}\ket{\nu_{2}^{M}}\approx\left\\{\begin{matrix}\ket{\nu_{2}^{M}}&,A^{(i)}_{CC}\gg
A^{R}_{CC}\\\
\cos{2\theta}\ket{\nu_{1}}+\sin{2\theta}\ket{\nu_{2}}&,A^{(i)}_{CC}\ll
A^{R}_{CC}.\end{matrix}\right.$ (13)

Given that the evolution is adiabatic, to get the survival and conversion
probabilities, we just have to replace the mass eigenstates in matter by their
respective ones in the vacuum and use the known mixing angle in vacuum
$\theta$. Therefore, we can summarize the solar neutrino survival probability
as follows:

$\langle
P_{\nu_{e}\rightarrow\nu_{e}}\rangle_{L}\approx\left\\{\begin{matrix}\frac{1}{2}-\frac{1}{2}\sin^{2}{2\theta}&,E<2\text{MeV}&\text{(Low
Energy)}\\\
\frac{1}{2}+\frac{1}{2}\cos{2\theta_{M}^{(i)}}\cos{2\theta}&,E>2\text{MeV}&\text{(Moderated
Energy)}\\\ \sin^{2}{\theta}&,E\gg 2\text{MeV}&\text{(High
Energy)},\end{matrix}\right.$ (14)

where we consider the probability averaged over propagated length $L$ due to
the uncertainty on its exact value.

####  4.1.2 Numerical Solution

For the numerical solution, we have adopted the mixing parameters $\Delta
m^{2}_{21}=7.53\times 10^{-5}\text{eV}^{2}$ and $\sin^{2}{\theta_{12}}=0.307$
(Zyla:2020zbs, ). For the electron density profile inside the Sun, we adopted
the following parametrization (Bahcall:2000nu, ):

$n_{e}(r)=245N_{A}exp(-10.45r/R_{\odot}),$ (15)

where $r$ is the radial distance from the center of the Sun, $N_{A}$ is the
Avogadro number and $R_{\odot}$ is the solar radius. The resulting outcome
from our numerical solver can be seen in figures 2 and 3 for high energy
neutrinos. In the figure, we can see the trajectory of the neutrino
polarization vector $\vec{P}$ and the resulting survival probability, which is
in agreement with the average one from the analytical solution. As expected,
in the high-energy case, the electron neutrino is created in the heavier mass
eigenstate $\ket{\nu_{e}}\approx\ket{\nu_{2}^{M}}$, which corresponds to a
$\vec{P}$ anti-parallel to $\vec{B}$ and remains in the same state, coupled to
$\vec{B}$, as the density decreases towards the vacuum. These results show
that our code and approach are in good agreement with the expected analytical
solutions, which allows us to proceed to the scenarios with neutrino-neutrino
interactions.

Figure 2: Numerical survival probability for high energy ($E_{\nu}=30$ MeV)
solar neutrinos in blue, and the analytical averaged probability in red.
Figure 3: Here we show the polarization vectors $\vec{P}$ (blue with
trajectory in yellow) and $\vec{B}$ (red), corresponding to the results shown
in figure 2. The left plot is a snapshot of the initial state, while the right
one corresponds to the final state outside the Sun (See the animation in
https://github.com/pedrodedin/Neutrino-Collective-
Effects/blob/main/Figures/Solar_High_Energy.gif).

###  4.2 1st Case - Isotropic and Mono-energetic Neutrino Gas

The first case with neutrino-neutrino interactions that we treat is the one
corresponding to an isotropic
($\left\langle\cos{\theta_{ij}}\right\rangle=0$), mono-energetic
($\omega_{i}=\omega$) and uniform neutrino gas composed of neutrinos with
opposed neutrino flavor isospin, also known as bipolar systems. That is the
case of a system composed of neutrinos with opposite polarization vectors or a
neutrino-antineutrino system of the same flavor. For simplicity, we will not
consider matter effects ($\lambda=0$). The presence of a uniform matter
potential will only change the mixing parameters to those effective in matter,
which does not change our conclusions. Regarding the vacuum mixing parameters,
we will use the following ones: $\Delta m^{2}_{31}=2.5\times 10^{-3}eV^{2}$
and $\sin^{2}{\theta_{13}}=2.1\times 10^{-2}$ Zyla:2020zbs , which are more
representative of a supernova scenario.

####  4.2.1 Symmetric System and Constant $\mu$

As an example of a bipolar system, we consider a scenario with an equal amount
of electron neutrinos and antineutrinos, which we call symmetric. We will also
consider the neutrino-neutrino potential $\mu$ as constant during the
evolution. Defining a total polarization vector for neutrinos as
$\vec{P}_{\nu}\equiv\sum_{i}^{N}\vec{P}_{\nu_{i}}$ and for antineutrinos as
$\vec{P}_{\overline{\nu}}\equiv\sum_{i}^{N}\vec{P}_{\overline{\nu}_{i}}$, the
evolution equation can be written as follows:

$\begin{split}\dot{\vec{P}}_{\nu}=\vec{P}_{\nu}\times\left[\omega\vec{B}+\mu(\vec{P}_{\nu}-\vec{P}_{\overline{\nu}})\right],\\\
\dot{\vec{P}}_{\overline{\nu}}=\vec{P}_{\overline{\nu}}\times\left[-\omega\vec{B}+\mu(\vec{P}_{\nu}-\vec{P}_{\overline{\nu}})\right].\end{split}$
(16)

We can further define a sum vector
$\vec{S}\equiv\vec{P}_{\nu}+\vec{P}_{\overline{\nu}}$ and a difference vector
$\vec{D}\equiv\vec{P}_{\nu}-\vec{P}_{\overline{\nu}}$ such that

$\begin{split}\dot{\vec{S}}=\omega\vec{D}\times\vec{B}+\mu\vec{D}\times\vec{S},\\\
\dot{\vec{D}}=\omega\vec{S}\times\vec{B}.\end{split}$ (17)

It is also useful to rewrite the sum vector as
$\vec{Q}\equiv\vec{S}-\frac{\omega}{\mu}\vec{B}$, so that in the limit of a
high neutrino-neutrino potential $\mu\gg\omega$ we have
$\vec{Q}\approx\vec{S}$. The evolution equations then become

$\begin{split}\dot{\vec{Q}}=\mu\vec{D}\times\vec{Q},\\\
\dot{\vec{D}}=\omega\vec{Q}\times\vec{B}.\end{split}$ (18)

As described in (Hannestad2006-zw, ), these are the same equations of a
pendulum attracted by a force field $\vec{F}=\omega\vec{B}$ , with angular
momentum $\vec{L}=\vec{D}$, length $\vec{r}=\vec{Q}$, and moment of inertia
$I=m|\vec{r}|^{2}=\mu^{-1}$. Therefore, the system composed of $\vec{P}_{\nu}$
and $\vec{P}_{\overline{\nu}}$ oscillates around $\omega\vec{B}$, resulting in
a oscillatory conversion probability 888For a more detailed description check
references (Hannestad2006-zw, ; Duan:2005cp, ). It is worth noting that the
sign of $\omega=\Delta m^{2}/2E$ depends on the neutrino mass hierarchy, such
that in the normal hierarchy ($\omega>0$) there is a small amplitude of
oscillation because $\vec{S}$ starts almost aligned with $\omega\vec{B}$. In
the case of the inverted hierarchy ($\omega<0$), $\vec{S}$ starts anti-aligned
with $\omega\vec{B}$, resulting in a complete amplitude of oscillation. This
analytical description corresponds exactly to what we find in our numerical
solutions shown in figures 4 and 5. As expected, for the normal hierarchy, the
vectors oscillate around $\vec{B}$, resulting in a small amplitude of
oscillation in the survival probability, while for the inverted hierarchy,
they oscillate around the opposite directions of $\vec{B}$, resulting in an
almost complete conversion amplitude.

Figure 4: Numerical survival probability for the symmetric and mono-energetic
neutrino gas with constant neutrino potential $\mu=1\times 10^{-8}$ eV and
energy $E=20$ MeV.

Figure 5: Snapshot of the polarization vectors corresponding to the results of
figure 4, where the left plot corresponds to the normal hierarchy and the
right to the inverted (See the animations in
https://github.com/pedrodedin/Neutrino-Collective-
Effects/blob/main/Figures/Bipolar_Oscillations_mu_constant_IH.gif and
https://github.com/pedrodedin/Neutrino-Collective-
Effects/blob/main/Figures/Bipolar_Oscillations_mu_constant_NH.gif).

####  4.2.2 Symmetric System and Varying $\mu$

A modification that we can do to the system above is to let the neutrino-
neutrino potential $\mu$ change over the neutrino trajectory. As a
representative of a supernova scenario we take the following profile for $\mu$
Hannestad2006-zw ; Mirizzi:2015eza , also shown in figure 6:

$\mu(r)=\left\\{\begin{matrix}\mu_{0}&r<R_{\nu}\\\
\mu_{0}\left(\frac{R_{\nu}}{r}\right)^{4},&r>R_{\nu}.\end{matrix}\right.$ (19)

Here, $R_{\nu}$ is the neutrinosphere, for which we take a typical value of
$10$ km, and $\mu_{0}=1\times 10^{-8}$ eV the initial neutrino potential.
Although we are using this supernova profile for $\mu$, given that our goal is
to explore supernova-like systems in the future, is worth remembering that we
still working on the case of an isotropic neutrino gas.

As discussed in the previous section, the bipolar system behaves like a
pendulum with a moment of inertia $I=m|\vec{r}|^{2}=\mu^{-1}$. Also, the
correspondent kinetic energy $|\vec{D}|^{2}/2I$ is conserved over a period of
oscillation Hannestad2006-zw . Therefore, as $\mu$ decreases the moment
inertia of the pendulum increases, and the oscillation is damped towards
$\omega\vec{B}$. This damping in the oscillation will lead to an almost total
conversion for the inverted mass hierarchy and practically no conversion for
the normal. Figure 7 shows our numerical results for this case. Again, the
numerical solutions fit quite well in the analytical description. As we can
see, the oscillation is damped towards total flavor conversion for the
inverted hierarchy, with a small amplitude of oscillation, and no conversion
for the normal hierarchy.

Figure 6: Profile used for the neutrino-neutrino potential $\mu$ as function
of the propagated distance $r$. The horizontal dashed lines shows the vacuum
frequencies $\omega$ for neutrinos with energies of 1 MeV and 40 MeV.

Figure 7: Survival probability for the inverted (left) and normal (right) mass
hierarchies considering the $\mu$ profile in figure 6 and a neutrino with
energy $E=20$ MeV (See the animation in
https://github.com/pedrodedin/Neutrino-Collective-
Effects/blob/main/Figures/Bipolar_Oscillations_mu_decreasing_IH.gif and
https://github.com/pedrodedin/Neutrino-Collective-
Effects/blob/main/Figures/Bipolar_Oscillations_mu_decreasing_IN.gif).

####  4.2.3 Asymmetric System and Varying $\mu$

The last scenario that we consider for our mono-energetic and isotropic gas is
the one with a different initial amount of neutrinos and anti-neutrinos, i.e.,
an asymmetrical system. This situation corresponds to a non-vanishing initial
$\vec{D}$, given by the initial difference of the lepton number. From the
equations of motion, $\vec{D}\cdot\vec{B}$ is conserved, which means that the
initial difference of the mass eigenstates is conserved. If the mixing angle
is small ($\theta\ll 1$), this conservation is equivalent to the conservation
of the net lepton number, which is the case for a supernova environment due to
the mixing angle suppression by the matter potential.

Figure 8 shows the survival probability of our numerical solution for a system
with a relative ratio of neutrinos to antineutrinos correspondent to
$1.0:0.7$. As expected, the initial net lepton number is conserved after a
period of oscillation for both mass hierarchies. We can also see the
phenomenon of synchronized oscillations for $r\lesssim 100$ km. This happens
when $\mu\gg\omega$ and $\vec{D}(0)\neq 0$ so that the precession of
$\vec{P}_{\nu}$ and $\vec{P}_{\overline{\nu}}$ is dominated by $\vec{D}$, with
precession frequencies of the order of $\mu$. Then, the vector $\vec{D}$
precesses around $\vec{B}$ with low oscillation frequency
$\omega_{sync}=\langle\omega\rangle$, given by the average of the vacuum
oscillation frequencies. In this scenario, all polarization vectors are
coupled to $\vec{D}$ and oscillate with the same frequency $\omega_{sync}$
around $\vec{B}$, where the name synchronized oscillations come from.

Figure 8: The same as figure 7 but considering a relative flux of $1.0$ for
neutrinos and $0.7$ for anti-neutrinos (See the animations in
https://github.com/pedrodedin/Neutrino-Collective-
Effects/blob/main/Figures/Bipolar_Oscillations_Assymetric_mu_decreasing_IH.gif
and https://github.com/pedrodedin/Neutrino-Collective-
Effects/blob/main/Figures/Bipolar_Oscillations_Assymetric_mu_decreasing_NH.gif).

###  4.3 2nd Case - Isotropic Neutrino Gas with Spectral Distribution

Finally, we will explore a neutrino isotropic gas with a spectral
distribution. In this case, we have equations that are similar to the mono-
energetic case, with the exception that now we have one polarization vector
for each momentum bin $\vec{p}_{i}$ and the neutrino-neutrino potential
emerges as a sum over all the momenta. If we define a new total difference
vector
$\vec{D}\equiv\sum_{\vec{p}_{2}}\vec{D}_{\vec{p}_{2}}=\sum_{\vec{p}_{2}}(\vec{P}_{\nu,\vec{p}_{2}}-\vec{P}_{\overline{\nu},\vec{p}_{2}})$,
we may write the evolution equations as follows:

$\begin{split}\dot{\vec{P}}_{\nu,\vec{p}_{1}}=\vec{P}_{\nu,\vec{p}_{1}}\times\left[\omega_{\vec{p}_{1}}\vec{B}+\sum_{\vec{p}_{2}}\mu(\vec{P}_{\nu,\vec{p}_{2}}-\vec{P}_{\overline{\nu},\vec{p}_{2}})\right]=\vec{P}_{\nu,\vec{p}_{1}}\times\left[\omega_{\vec{p}_{1}}\vec{B}+\mu\vec{D}\right],\\\
\dot{\vec{P}}_{\overline{\nu},\vec{p}_{1}}=\vec{P}_{\overline{\nu},\vec{p}_{1}}\times\left[-\omega_{\vec{p}_{1}}\vec{B}+\sum_{\vec{p}_{2}}\mu(\vec{P}_{\nu,\vec{p}_{2}}-\vec{P}_{\overline{\nu},\vec{p}_{2}})\right]=\vec{P}_{\overline{\nu},\vec{p}_{1}}\times\left[-\omega_{\vec{p}_{1}}\vec{B}+\mu\vec{D}\right].\end{split}$
(20)

As we did for the mono-energetic case, we can write the evolution equation for
the sum $\vec{S}_{\vec{p}_{1}}$ and difference $\vec{D}_{\vec{p}_{1}}$
vectors, which now depend on the momentum $\vec{p}_{1}$:

$\begin{split}\dot{\vec{S}}_{\vec{p}_{1}}=\omega_{\vec{p}_{1}}\vec{D}_{\vec{p}_{1}}\times\vec{B}+\mu\vec{D}\times\vec{S}_{\vec{p}_{1}},\\\
\dot{\vec{D}}_{\vec{p}_{1}}=\omega_{\vec{p}_{1}}\vec{S}_{\vec{p}_{1}}\times\vec{B}.\end{split}$
(21)

If $\mu\gg\omega_{\vec{p}_{1}}$ for all modes, all $\vec{S}_{\vec{p}_{1}}$
evolve likewise, resulting in an evolution similar to the mono-energetic case
for each mode (Hannestad2006-zw, ). However, if we consider a decreasing
$\mu$, the resulting asymptotic survival probabilities will be different for
each mode. As discussed in section 4.2.3, for a small mixing angle, the
equations of motion lead to the conservation of the net lepton number.
Therefore, if we have more neutrinos than antineutrinos and consider the
inverted hierarchy scenario 999In the case of the normal hierarchy we do not
have any relevant conversion, as discussed for the mono-energetic case., we
will have sufficient lepton number to completely convert the antineutrinos but
only a fraction of the neutrinos, so that we have the following law for flavor
conservation:

$\int_{E_{split}}^{\infty}dE\left[\phi_{\nu_{e}}(E)-\phi_{\nu_{x}}(E)\right]=\int_{0}^{\infty}dE\left[\phi_{\overline{\nu}_{e}}(E)-\phi_{\overline{\nu}_{x}}(E)\right],$
(22)

where $E_{split}$ is the energy above which total conversion occurs and none
below it, leading to the phenomenon called spectral split (Fogli_2007, ). As
we can see, this energy depends only on the initial net lepton number, which
will be conserved through the propagation. For a deeper discussion on the
phenomenology of the spectral split, we refer the reader to other references
(Raffelt2007-xu, ; Duan2006-dp, ). Here, we will limit ourselves to using
equation (22) to analyze our numerical results.

####  4.3.1 Numerical Solution

To test this scenario in our numerical code, we decided to use the following
parametrization corresponding to a supernova emission (Keil_2003, ;
Keil:2003sw, ):

$f_{\alpha}(E)=N\left(\frac{E}{\overline{E}}\right)^{\alpha}e^{-(\alpha+1)E/\overline{E}},$
(23)

where $\overline{E}$ is the averaged energy, $\alpha$ is the pinching
parameter, and $N$ is a normalization constant. To implement it in the
numerical code, we discretize the spectrum with an energy bin size of $\Delta
E=0.5$ MeV. Our results are shown in figures 9 and 10 for the inverted and
normal mass hierarchy respectively. As we can see, for the normal hierarchy,
there is no conversion at all, with the exception of some low-energy
conversions (Fogli:2008pt, ). For the inverted hierarchy we have a total
conversion in the antineutrino sector and a spectral split in the neutrino
one, given that there is no sufficient lepton number to complete the
conversion.

Figure 9: Initial (dashed lines) and final (solid lines) spectra for the
inverted mass hierarchy, with mean energies $\overline{E}_{\nu_{e}}=10$ MeV,
$\overline{E}_{\overline{\nu}_{e}}=15$ MeV, $\overline{E}_{\nu_{x}}=24$ MeV
and total energies
$\varepsilon_{\nu_{e}}=\varepsilon_{\overline{\nu}_{e}}=\varepsilon_{\nu_{x}}$.
Figure 10: Initial (dashed lines) and final (solid lines) spectra for the
normal mass hierarchy, with mean energies $\overline{E}_{\nu_{e}}=10$ MeV,
$\overline{E}_{\overline{\nu}_{e}}=15$ MeV, $\overline{E}_{\nu_{x}}=24$ MeV
and total energies
$\varepsilon_{\nu_{e}}=\varepsilon_{\overline{\nu}_{e}}=\varepsilon_{\nu_{x}}$.

Although we are working with an isotropic neutrino gas, the spectral swap is
expected to happen in some non-isotropic scenarios. For example, if we
consider a supernova bulb model of emission with a single-angle approximation,
the evolution equation becomes similar to the isotropic one, except for a
global geometrical $D(r)$ that decreases with radius (Duan2006-dp, ). However,
if multi-angle emission is considered, we may expect some decoherence due to
the different phases along different trajectories (Hannestad2006-zw, ). In any
case, we leave the topic of non-isotropic emissions for future work.

##  5 Conclusions

In this paper, we have described a code to find numerical solutions for the
neutrino evolution in cases where neutrino-neutrino forward scattering is
relevant, which leads to the so-called collective effects. We introduced the
equations to be solved and how they were implemented numerically. Then, we
have shown the application of our code for different cases and compared it
with other works. First, we explored the consistency of our approach by
working with a solar neutrino scenario, which has shown to be in good
agreement with the analytical solutions. After that, we explored the scenario
of an isotropic gas of neutrinos, starting with a mono-energetic case composed
of $\nu_{e}$ and $\overline{\nu}_{e}$. In this case, we explored different
configurations for the relative amount of $\nu_{e}$ and $\overline{\nu}_{e}$,
and different behaviors for $\mu$ (constant or decreasing). Finally, we
explored a scenario with a spectral distribution for the neutrinos, where we
saw the phenomena of spectral split.

Our intention with this work was not only to show our results in the effort to
understand collective effects but especially to provide an open-source
repository with a pedagogical description of these effects. We were motivated
to do this due to the lack of open-source codes in this research field, even
with a significant number of papers that rely on numerical results. In our
understanding, this lack of transparency hampers reproducibility, a
cornerstone of science. Therefore, we hope that this paper encourages future
works to make their codes open-source or in any equivalent way to facilitate
reproducibility. Besides that, we plan to continue developing the code
presented here, extending it to more supernova-like scenarios. In our future
work, we intend to explore non-isotropic models, such as the Bulb model
Duan2006-dp , and the relatively new phenomenon of fast oscillations
Tamborra:2020cul .

###### Acknowledgements.

This work was supported by São Paulo Research Foundation (FAPESP) grants no.
2019/08956-2 and no. 14/19164-6. The author is thankful to Marcos V. dos
Santos for useful discussions during the production of this article.

## References

  * (1) S. Hannestad, G. G. Raffelt, G. Sigl and Y. Y. Y. Wong, _Self-induced conversion in dense neutrino gases: Pendulum in flavour space_ , .
  * (2) H. Duan, G. M. Fuller and Y.-Z. Qian, _Collective neutrino flavor transformation in supernovae_ , _Phys. Rev. D_ 74 (2006) 123004, [astro-ph/0511275].
  * (3) H. Duan, G. M. Fuller, J. Carlson and Y.-Z. Qian, _Simulation of coherent nonlinear neutrino flavor transformation in the supernova environment: Correlated neutrino trajectories_ , _Phys. Rev. D_ 74 (Nov., 2006) 105014.
  * (4) H. Duan, G. M. Fuller and Y.-Z. Qian, _Collective neutrino oscillations_ , .
  * (5) A. Mirizzi, I. Tamborra, H.-T. Janka, N. Saviano, K. Scholberg, R. Bollig et al., _Supernova Neutrinos: Production, Oscillations and Detection_ , _Riv. Nuovo Cim._ 39 (2016) 1–112, [1508.00785].
  * (6) I. Tamborra and S. Shalgar, _New Developments in Flavor Evolution of a Dense Neutrino Gas_ , _Ann. Rev. Nucl. Part. Sci._ 71 (2021) 165–188, [2011.01948].
  * (7) R. F. Sawyer, _The multi-angle instability in dense neutrino systems_ , _Phys. Rev. D_ 79 (2009) 105003, [0803.4319].
  * (8) L. Wolfenstein, _Neutrino oscillations in matter_ , _Physical Review D_ 17 (1978) 2369.
  * (9) P. D. Neto and E. Kemp, _Neutrino-(anti)neutrino forward scattering potential for massive neutrinos at low energies_ , 2111.11480.
  * (10) G. Sigl and G. Raffelt, _General kinetic description of relativistic mixed neutrinos_ , _Nuclear Physics B_ 406 (1993) 423–451.
  * (11) E. Akhmedov, _Neutrino oscillations in matter: from microscopic to macroscopic description_ , _JHEP_ 02 (2021) 107, [2010.07847].
  * (12) P. D. Neto, _pedrodedin/Neutrino-Collective-Effects: First Release - Isotropic Neutirno Gas_ , Feb., 2022.  10.5281/zenodo.5964385.
  * (13) P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau et al., _SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python_ , _Nature Methods_ 17 (2020) 261–272.
  * (14) S. Mikheev and A. Y. Smirnov, _Resonance amplification of oscillations in matter and spectroscopy of solar neutrinos_ , _Yadernaya Fizika_ 42 (1985) 1441–1448.
  * (15) Particle Data Group collaboration, P. Zyla et al., _Review of Particle Physics_ , _PTEP_ 2020 (2020) 083C01.
  * (16) J. N. Bahcall, M. H. Pinsonneault and S. Basu, _Solar models: Current epoch and time dependences, neutrinos, and helioseismological properties_ , _Astrophys. J._ 555 (2001) 990–1012, [astro-ph/0010346].
  * (17) G. Fogli, E. Lisi, A. Marrone and A. Mirizzi, _Collective neutrino flavor transitions in supernovae and the role of trajectory averaging_ , _Journal of Cosmology and Astroparticle Physics_ 2007 (dec, 2007) 010–010.
  * (18) G. G. Raffelt and A. Y. Smirnov, _Self-induced spectral splits in supernova neutrino fluxes_ , .
  * (19) M. T. Keil, G. G. Raffelt and H. Janka, _Monte carlo study of supernova neutrino spectra formation_ , _The Astrophysical Journal_ 590 (Jun, 2003) 971–991.
  * (20) M. T. Keil, _Supernova neutrino spectra and applications to flavor oscillations_ , other thesis, 8, 2003.
  * (21) G. L. Fogli, E. Lisi, A. Marrone, A. Mirizzi and I. Tamborra, _Low-energy spectral features of supernova (anti)neutrinos in inverted hierarchy_ , _Phys. Rev. D_ 78 (2008) 097301, [0808.0807].


# SoMoFormer: Multi-Person Pose Forecasting with Transformers

Edward Vendrow1, Satyajit Kumar1, Ehsan Adeli1, and Hamid Rezatofighi2
1Stanford University, 2Monash University
{evendrow<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Human pose forecasting is a challenging problem involving complex human body
motion and posture dynamics. In cases that there are multiple people in the
environment, one’s motion may also be influenced by the motion and dynamic
movements of others. Although there are several previous works targeting the
problem of multi-person dynamic pose forecasting, they often model the entire
pose sequence as time series (ignoring the underlying relationship between
joints) or only output the future pose sequence of one person at a time. In
this paper, we present a new method, called Social Motion Transformer
(SoMoFormer), for multi-person 3D pose forecasting. Our transformer
architecture uniquely models human motion input as a joint sequence rather
than a time sequence, allowing us to perform attention over joints while
predicting an entire future motion sequence for each joint in parallel. We
show that with this problem reformulation, SoMoFormer naturally extends to
multi-person scenes by using the joints of all people in a scene as input
queries. Using learned embeddings to denote the type of joint, person
identity, and global position, our model learns the relationships between
joints and between people, attending more strongly to joints from the same or
nearby people. SoMoFormer outperforms state-of-the-art methods for long-term
motion prediction on the SoMoF benchmark as well as the CMU-Mocap and
MuPoTS-3D datasets. Code is available at blinded.

##  1 Introduction

Forecasting human motion is a challenging problem with applications in
robotics [12], computer graphics [14], and tracking in the context of
autonomous vehicles [10]. Any system requiring understanding or navigation of
multi-person environments requires reasoning about how people will move to
make reliable operational decisions.

To date, most of the human skeleton pose forecasting methods produce
predictions for one person at a time [6, 20, 19, 21, 23, 30, 29], ignoring the
global translation and social context of other people in a scene. Simultaneous
research into trajectory forecasting [13, 9, 27, 15, 32, 17] models an entity
as a single 2D point on a ground plane, which provides useful global
translation prediction, but is insufficient for tasks requiring detailed body
pose information. Recent methods [1, 2] attempt to model social interactions
through pooling and graph-based methods to capture social dependencies for
multi-person forecasting, but fall short on long time-horizons with recurrent
architectures that quickly accumulate errors. Subsequent methods built on the
success of attention-based models for long-term motion prediction [23, 9],
with [23] circumventing the sequential nature of time-series pose forecasting
with a Transformer architecture capable of predicting an entire single-person
pose trajectory in parallel. Yet, since their model used a fixed input size
with a time-series input, generalization to the multi-person setting is
difficult.

Figure 1: Visualization of SoMoFormer predictions on the 3DPW dataset for two
sequences, with the ground-truth future sequence (GT) on top and predicted
sequence (PR) below. SoMoFormer predicts realistic human motion, closely match
the ground-truth even in a multi-person scenario. Figure 2: SoMoFormer uses a
Transformer Encoder, which accepts each input token as the DCT-encoded
(discrete cosine transform) padded trajectory of one joint. SoMoFormer learns
a embedding for joint type and person identity which are added to the token,
and then a grid position embedding encoding global position for each person is
concatenated. SoMoFormer is able to predict pose trajectories for multiple
people simultaneously while leveraging attention over joints to model human
body dynamics.

To our knowledge, only [30] propose a multi-person Transformer for pose
forecasting. Their approach uses both a local and global encoder to model
temporal and social dependencies respectively, which limits them to predict
poses just one person in the scene at a time, requiring multiple passes to
produce a multi-person scene. This limitation affects all prior work including
those that use social pooling methods. Furthermore, [30] address the problem
of a variable number of people by using a global encoder with one input query
for each frame for each person, which then introduces the challenge of
multiple input queries corresponding to the same timestep. Indeed, all prior
attention-based methods structure pose as a time series input, making
extensions to multi-person cases difficult because they require the model to
predict over an input containing both time-series and person-series sequences.

In this work, we introduce SoMoFormer, an attention-based pose forecasting
model which unifies multi-person pose forecasting (See Figs. 1 and 2).
Different from the previous methods, we reformulate the problem of pose
forecasting from predicting a time-sequence series of poses to predicting a
joint-sequence series of trajectories where each input query denotes the
trajectory of a skeleton joint coordinate over the entire input timeframe. Our
re-framing of the pose forecasting problem allows us to design SoMoFormer,
which accepts a set of skeleton joint trajectories as input rather than a set
of pose timesteps, enhancing interactions among joints and allowing our model
to predict an entire future sequence without recurrence. Crucially, this
architecture naturally extends to multi-person scenes by simply adding more
joint trajectories to the input query set for all people in the scene.
SoMoFormer enables this new type of motion prediction by using a learned
embedding to denote which person each joint belongs to as well as learned
embeddings for joint type and location within the scene, to enable the
prediction of a future multi-person pose sequence in one pass. Experimental
results on multiple datasets show that SoMoFormer is superior at learning the
relationships between joints and between people, and consequently outperforms
all previous state-of-the-art methods in human pose forecasting. In summary,
our contributions include:

  * •

We introduce a novel transformer-based method, named SoMoFormer, which uses
joint-aware attention for multi-person human pose forecasting.

  * •

We reformulate the problem of pose forecasting by accepting and predicting a
query for each joint rather than a query for each timestep, allowing
SoMoFormer to perform long-term pose forecasting for multiple people without
recurrence.

  * •

SoMoFormer outperforms state-of-the-art methods for long-term motion
prediction on the SoMoF benchmark 111The SoMoF benchmark:
https://somof.stanford.edu/, as well as on the CMU-Mocap [18] and MuPoTS-3D
[24] datasets.

  * •

Analysis of our model attention weights reveals that SoMoFormer learns rich
representations of human motion and dynamics.

##  2 Related Work

Human Trajectory Forecasting: Many recent works model human motion as a
trajectory, representing each person as a 2-dimensional point on a plane [13,
9, 27, 15, 32, 17]. [27] use RNNs combined with an attentive GAN to provide
socially plausible output predictions. [13] use a graph attention network
combined with a GAN to encode the social interactions between humans in the
scene. Catalyzed by the success of the Transformer model [28], many recent
works have used attention-based methods, such as transformers, to better model
sequential data and complex human motion relationships [20, 19, 21, 2, 9, 17,
2, 23]. Within trajectory forecasting, [9] used a transformer to model the
trajectory of each person through time. [17] improved upon this by proposing a
stacked transformer that is able to use the motion from multiple objects in
the scene. Nevertheless, trajectory forecasting methods fail to capture the
complex pose dynamics involved in human motion.

Single-Person Pose Forecasting: Single-person human pose forecasting methods
predict the future joint coordinates without global translation. Early single-
person pose forecasting models employed Hidden Markov Models [4], linear
dynamics models [26], and hand-crafted features such as hard constraints on
limb length and joint angles [3, 31]. Later methods moved toward end-to-end
neural networks with few or no hand-crafted features. Many of these recent
methods use an RNN backbone, with some using graph attention networks or GANs
to extend the prediction to multiple entities or provide a plausible output.
Chiu et al. [6] use a hierarchical RNN to predict human motion. [20] models
human motion as a graph of encoded motions for each joint coordinate with a
GNN architecture to pass information between nodes. Wang et al. [29] extend
this work by compressing the node representation to all dimensions of a joint.
Mao et al. [19] introduce an attention-based model to capture the similarities
between current and historical motion sub-sequences, allowing the model to
aggregate past motions for long-term prediction. Cai et al. [5] use a human
skeleton model to propagate joint predictions based on kinematic connections.
Matínez-González et al. [23] used an encoder-decoder architecture to
simultaneously decode all future timesteps without recurrence. All of the
above methods remove global translation, modeling just local human skeletal
motion. However, in a multi-person setting, such as the environment around an
autonomous vehicle, global interactions between people must also be taken into
account when making a prediction.

Multi-Person Pose Forecasting: With the need to model scenes with multiple
people, some recent works focus on multi-person forecasting to predict the
future for an entire scene. As with both pose and trajectory forecasting,
these works similarly adopted attention-based methods. Adeli et al. [2] use
graph attentional networks to model interaction between humans and objects,
but use an RNN to predict future motion. Matínez-González et al. [23] use a
transformer to predict an entire future sequence without recurrence, but use
time-series queries making multi-person prediction infeasible. Wang et al.
[30] use a transformer-based architecture to consider the global interactions
between multiple people, but can only make inferences for one person at a
time. Additionally, each input token for Wang et al. represents the position
of all joints at a single timestep, making an extension to multi-person
modeling unnatural. We propose a method to perform unified multi-person pose
forecasting without recurrence by performing attention over a set of joints
rather than a set of timesteps. Our method is able to use the attention
mechanism of a transformer to learn the relationships between joints and
between people simultaneously to naturally model human body dynamics and
multi-person interactions.

##  3 Methods

Given a scene with $N$ humans in motion, each composed of $J$ joints, we wish
to predict their motion for some number of future timesteps. Formally, for
each person we are given the pose history sequence
$X^{n}_{1:t}=[x^{n}_{1},x^{n}_{2},\ldots,x^{n}_{t}]$ where $n=1,\ldots,N$
denotes the person, $t$ represents the input pose sequence length, and each
$x_{k}^{n}=(j_{1},j_{2},\ldots,j_{J})\in\mathbb{R}^{J\times 3}$ represents
each pose of person $n$ at time $k$ as the three-dimensional Cartesian
coordinates of each joint, in global coordinates. We aim to build a model to
predict motion for each person for $T$ timesteps into the future, predicting
the pose sequence $S^{n}_{t+1:T}$.

###  3.1 Proposed Architecture

Figure 2 is an overview of our proposed pose forecasting framework. Our model
is a transformer encoder that accepts a token for each coordinate ($x$, $y$,
and $z$) of every joint in the scene, encoded as the padded coordinate
sequence over the input timeframe and transformed via DCT. Our model performs
trajectory completion for $(X^{n}_{1:t}||\hat{X}^{n}_{t+1:T})$, where $||$
denotes concatenation, giving us the concatenation of the known history
sequence $X^{n}_{1:t}$ and padded sequence $\hat{X}^{n}_{t+1:T}$ generated
using the previous history. By using a padded sequence, our input and output
queries are the same length, allowing the model to predict the difference
between the DCT representation of the input and output instead of generating a
full DCT sequence. We use a constant-padded sequence rather than a random or
fixed padding value because seeding the output with a reasonable
initialization makes the prediction task simpler. Taking advantage of residual
connections within the transformer, our model refines the predicted motion at
each layer. Differing from previous multi-person pose forecasting methods [30]
our model accepts one input query for each joint, rather than for each
timestep. Thus our model accepts $N\times J\times 3$ input queries in total
for a scene with $N$ people, where each input query is a vector of size
$(t+T)$ containing one cartesian dimension of a joint trajectory over all
$(t+T)$ timesteps.

To improve generalization to large scenes or distant people, our model accepts
joint queries with global translation removed by subtracting the neck or
pelvis joint coordinate of the last-known ($t$-th) frame for every person.
This translation is added back afterward to recover the global coordinate
position. Since the input queries for each person do not have global
translation information, the global position of each person is instead encoded
via a learned embedding, and concatenated with the joint trajectory
representation. When a scene has multiple people, our model accepts queries
for all people in a scene. We encode both joint type (left knee, right
shoulder, etc.) and personal identity with learnable embeddings which are
added to the joint queries.

###  3.2 Discrete Cosine Transform

Figure 3: Left: We randomly rotate poses around the y-axis at training time.
The red, blue, and green pose trajectories are all rotations of the same pose
sequence. Right: We randomly permute the order of people so that the learned
identity embedding sees a more diverse set of training examples. Each color
represents joints from a different person or a masked query. We keep joints
from the same person together so that the identity embedding is consistent.

Predicting Cartesian coordinates directly does not take advantage of human
motion being continuous, smooth, and somewhat periodic. Instead, recent works
[19, 30] use a Discrete Cosine Transform (DCT) to encode human motion into the
frequency domain as a collection of coefficients. In addition to more
reasonably encoding human motion, using DCTs allows us to predict the entire
trajectory for all future frames simultaneously.

For a motion sequence of a single coordinate $(x_{1},\ldots,x_{T})$, the
$l$-th DCT coefficient is computed by

$\displaystyle
C_{l}=\sqrt{\frac{2}{T}}\sum_{t=1}^{T}\frac{x_{t}}{\sqrt{1+\delta_{l1}}}cos\frac{\pi}{2T}(2t-1)(l-1)$
(1)

for

$\displaystyle\delta_{ij}=\begin{cases}1&\text{if $i=j$}\\\ 0&\text{if $i\neq
j$}\end{cases}$ (2)

where $l=1,2,\ldots,T$, yielding the same number of coefficients as the
sequence length. This process gives the coefficient sequence
$(C_{1},\ldots,C_{T})$ which is used as an input token to the model for the
corresponding joints coordinate to predict the final, completed trajectory
$(\tilde{C}_{1},\ldots,\tilde{C}_{T})$. Given this coefficient sequence, we
can recover the coordinates using the inverse DCT, where we get a specific
coordinate via

$\displaystyle\tilde{x}_{t}=\sqrt{\frac{2}{T}}\sum_{l=1}^{T}\frac{\tilde{C}_{l}}{\sqrt{1+\delta_{l1}}}cos\frac{\pi}{2T}(2t-1)(l-1).$
(3)

In practice, this DCT encoding is easy to use as it can be implemented with a
simple invertible linear transformation and significantly improves results.

###  3.3 Person and Joint Embeddings

Since the input into a Transformer is position invariant, Transformers use
embeddings to denote the input query order, generally in the form of a vector
that is added or concatenated with the input query. In natural language
processing and other time series methods, a common positional embedding
include alternating pairs of sines and cosines to denote the input order [7].
Similarly, for each input joint query, our model should encode both its joint
type (neck, hip, shoulder, etc.) as well as the identity of the person it
corresponds to. We accomplish this by introducing the use of two distinct
embeddings denoting joint type and person identity. For each input joint
query, we add both an embedding corresponding to the joint type, and an
embedding corresponding to the person’s identity. Both embeddings are
randomly-initialized and learned at training time.

###  3.4 Grid Positioning

Since we subtract global translation in the model input queries for easier
training, we use a grid positioning method to provide global translation
information to the model. Inspired by the Vision Transformer [8] which divides
an image into a patch grid and encodes patch placement with positional
embedding, we divide the global scene into a grid of cells and assign each
cell a randomly-initialized, learnable positional embedding, which is
concatenated to the input query. Each person in a scene is assigned to a cell
based on the position of the neck joint at the last known frame. The model
learns to associate the grid embedding to the distance between different
people in a scene.

###  3.5 Random First-Frame Selection

Our training datasets often consist of extended scenes with many frames. We
sample a subsequence from the entire longer sequence by randomly selecting a
start frame and taking the pose trajectory over the next $t+T$ frames, which
increases the diversity of training examples as compared to simply dividing
each scene into fixed subsequences. Like in [29], we further augment our data
by using reversed pose sequences as additional training data.

###  3.6 Random Rotation

At training time, we randomly rotate scenes about the vertical axis on a
uniform distribution within $[0,2\pi]$, creating scenes that are equally
plausible. This data augmentation method helps prevent the model from
overfitting to a specific trajectory. The left diagram in Figure 3 visualizes
a pose sequence under several rotations. Note that each of the rotated pose
sequences is valid and useful. Our use of random rotation is crucial to the
effectiveness of a learned grid embedding because it helps expose every grid
cell to a variety of scenes.

###  3.7 Random Person Permutation

Since the order in which people are input into the model should not matter, we
randomly permute this order at training time, as described in Figure 3, to
make our learned identity embedding more robust. Our training data consists of
scenes with different numbers of people, requiring us to use a padding mask
with our Transformer model. In these cases, we also permute non-padded people
across the elements of the padding mask as depicted in the figure. This
technique helps prevent overfitting in which the model could associate
specific poses with specific identity embeddings. Additionally, by permuting
padded examples, we allow all identity embedding to see an equal share of non-
padded training examples.

##  4 Experiments

This section details the experiments we performed to evaluate our approach.

Table 1: Experimental results in VIM on the SoMoF 3DPW test set. | 3DPW Prediction in Time |
---|---|---
Method | 100ms | 240ms | 500ms | 640ms | 900ms | Overall
Mo-Att [19] \+ ST-GAT [11] | 62.4 | 94.6 | 153.2 | 188.0 | 249.9 | 149.6
SC-MPF [1] | 45.4 | 73.7 | 129.2 | 159.5 | 208.3 | 123.2
Zero Velocity | 29.4 | 53.6 | 94.5 | 112.7 | 143.1 | 86.7
TRiPOD [2] | 31.0 | 50.8 | 84.7 | 104.1 | 150.4 | 84.2
DViTA [25] | 19.5 | 36.9 | 68.3 | 85.5 | 118.2 | 65.7
FutureMotion [30] | 9.5 | 22.9 | 50.9 | 66.2 | 97.4 | 49.4
SoMoFormer | 9.1 | 21.3 | 47.5 | 61.6 | 91.9 | 46.3
Table 2: Experimental results in MPJPE on CMU-Mocap and MuPoTS-3D. | CMU-Mocap Test Set | MuPoTS-3D Test Set
---|---|---
Method | 1 sec | 2 sec | 3 sec | 1 sec | 2 sec | 3 sec
HRI [19] | 0.503 | 0.932 | 1.422 | 0.261 | 0.469 | 0.714
LTD [20] | 0.480 | 0.869 | 1.181 | 0.191 | 0.337 | 0.466
Multi-Range [30] | 0.456 | 0.840 | 1.11 | 0.206 | 0.393 | 0.574
SoMoFormer | 0.42 | 0.80 | 1.06 | 0.173 | 0.305 | 0.423

###  4.1 Metrics

MPJPE.  Consisted with prior work [23, 19, 30], we report our Mean Per Joint
Position Error (MPJPE), which measures the average Euclidean distances between
ground-truth and predicted joint positions. We use global coordinates to
better evaluate when the predicted trajectory deviates from the ground truth.

Visibility-Ignored Metric (VIM).  First introduced in [2], VIM calculates the
mean $3J$-dimensional distance between the ground-truth and predicted joint
positions after flattening together the joint and coordinate dimensions. This
is the metric used for the SoMoF Benchmark.

###  4.2 Data

We use the following datasets to train our model. These datasets provide human
body meshes, which are regressed to produce joint coordinates.

3DPW [22].  We used the 3D Poses in the Wild Dataset (3DPW) in our experiments
for human motion prediction. This dataset provides human motion sequences in
real-world settings, with over 60 video sequences. Since we used the SoMoF
benchmark to evaluate our model, we use the SoMoF benchmark splits for 3DPW in
which the 3DPW train and test set are flipped. This means that we technically
train our model using the 3DPW test set, and evaluate on the 3DPW train set.

AMASS [18]. The Archive of Motion Capture As Surface Shapes (AMASS) provides a
massive dataset of human motion capture sequences with over 40 hours of motion
and 11,000 motions provided as SMPL mesh models. During training, we use the
CMU, BMLMovi, and BMLRub subsets of this dataset, which provide large-scale
and varied sets of motions. Since many of these sequences are single-person,
we synthesize additional training data by mixing together sampled sequences to
create multi-person training data.

CMU-Mocap. The Carnegie Mellon University Motion Capture Database (CMU-Mocap)
provides motion capture recording from 140 subjects performing various
activities. We use training and testing sets derived by Wang et al. [30] to
train and evaluate our model.

MuPoTS-3D [24]. The Multi-person Pose estimation Test Set in 3D (MuPoTS-3D)
provides 8,000 annotated frames of poses from 20 real-world scenes. We use
this test dataset to evaluate our model performance.

###  4.3 Training

Table 3: Ablation study on the SoMoF 3DPW validation set, with results shown in VIM (top) and MPJPE (bottom). Note that this 3DPW validation set is different from the 3DPW test set in Table 1. Method | 100ms | 240ms | 500ms | 640ms | 900ms | Overall
---|---|---|---|---|---|---
Baseline | 9.0 | 23.3 | 54.5 | 68.9 | 95.8 | 50.3
\+ Learned Embedding | 8.6 | 22.0 | 50.3 | 63.6 | 92.3 | 47.4
\+ Data Augmentation | 8.0 | 20.3 | 46.2 | 58.4 | 84.3 | 43.5
\+ Grid Positioning | 7.6 | 19.7 | 44.6 | 55.9 | 80.2 | 41.6
Baseline | 2.0 | 5.2 | 12.7 | 16.4 | 23.6 | 12.0
\+ Learned Embedding | 1.9 | 4.9 | 11.9 | 15.6 | 23.4 | 11.5
\+ Data Augmentation | 1.8 | 4.5 | 11.0 | 14.4 | 21.4 | 10.6
\+ Grid Positioning | 1.7 | 4.4 | 10.6 | 13.7 | 20.2 | 10.1

Rather than simply taking the loss between the outputs of the transformer and
the ground truth, we further encourage our model to avoid overfitting and
learn meaningful motion representations at each layer by taking an auxiliary
loss on the output of every layer. Intuitively, this allows the model to
iteratively refine the predicted motion at each layer. Given a dataset with
$M$ training examples and an $L$-layer, $S^{n}_{1:t+T}\in\mathbb{R}^{Jx3}$
denotes the ground-truth 3D coordinates and$S^{n}_{\ell,1:t+T}$ denotes the
predicted 3D coordinates at the $\ell$-th layer. We use $L_{2}$ loss to
minimize the error between the ground truth and predicted coordinates:

$\displaystyle\mathcal{L}=\sum_{\ell=1}^{L}\sum_{i=1}^{M}\sum_{n=1}^{N}\lambda_{L}\|S^{n}_{t+1:t+T}-\hat{S}^{n}_{L,t+1:t+T}\|^{2}$
(4)

where $\lambda_{\ell}$ denotes the weight of the loss from the $\ell$-th
layer.

###  4.4 Implementation Details

Our transformer encoder has 6 layers, 8 attention heads, and a hidden
dimension of 1024. We train for 256 epochs with a batch size of 512. We use
the Adam optimizer with an initial learning rate of 0.001, which is decayed by
0.1 at the end of training. We use $\lambda_{6}=1,\lambda_{1...5}=0.2$. We
encode global position using a grid embedding by dividing the scene into 5x5
grid, corresponding to 25 unique grid embeddings. SoMoFormer is developed
using PyTorch.

##  5 Experimental Results

We first show that our method outperforms previous approaches in human pose
forecasting on the SoMoF 3DPW, CMU-Mocap, and MuPoTS-3D test sets. Then, we
perform an ablation study to provide insight into our model and training
methods. Using visualized prediction, we qualitatively validate the
performance of our model. Finally, we visualize the attention graphs from our
model to demonstrate that it learns meaningful human body motion
representations.

###  5.1 Results on SoMoF Benchmark

The SoMoF benchmark [1, 2] provides a benchmark for multi-person human pose
trajectory. Each sequence has 16 frames (1070 ms) of input to predict the next
14 frames (930 ms), where each frames consists of joints positions for
multiple people. Results are reported as the mean VIM at multiple future
timesteps. Just as [29], we train on the 3DPW [22] and AMASS [18] datasets,
which provide both multi-person and single-person data, and finally we
finetune on 3DPW. Since SoMoF only uses 13 joints for evaluation, we use just
these joints during training as well. We report in Table 1 a comparison of
methods on the SoMoF 3DPW test set 222Our submission (currently anonymous) to
the SoMoF benchmark is dated March 7, 2022.. Our model consistently
outperforms all previous methods.

###  5.2 Results on CMU-Mocap and MuPoTS-3D

We additionally compare our method to Wang et al. [30] who proposed a recent
method for multi-person pose forecasting achieving state-of-the-art results on
several datasets, as well as other recent methods including HRI [19] and LTD
[20]. Using their protocols, models are trained on a synthesized dataset
mixing sampled motions from CMU-Mocap to create 3-person scenes, and evaluated
on both CMU-Mocap and MuPoTS-3D. We give 15 frames (1000 ms) of history as
input to predict the next 45 frames (3000 ms), and report the MPJPE at 1, 2,
and 3 seconds in the future. For fair comparison, we train and evaluate each
method using the code and data provided by [30]. We report in Table 2 a
comparison of each method on the testing datasets. Our model consistently
outperforms the others on both CMU-Mocap and MuPoTS-3D. We observe a
significant difference in performance on MuPoTS-3D, which unlike CMU-Mocap is
not synthesized from a combination of samples poses, meaning that the data
represents real social interactions. While [30] predict the future pose
forecast for just one person at a time, SoMoFormer is able to fully leverage
attention over joints from all people in a scene to make predictions for all
people simultaneously. When data contains interactions between multiple
people, as in MuPoTS-3D, our method can better model these interactions to
predict the future.

Figure 4: A plot of the distance between two people against the attention
score in SoMoFormer, averaged across joints. Each sample (blue) is drawn from
3-person scenes in CMU-Mocap, and we draw the best-fit line (red). The closer
two people are, the higher the attention score between them.

###  5.3 Ablation Study

SoMoFormer introduces several components in order to achieve state-of-the-art
performance: learned joint type and identity embeddings, data augmentation,
and grid positioning. We perform an ablation study on each of these components
to examine their contribution to SoMoFormer in Table 3. Each model is trained
on the SoMoF 3DPW training set and evaluated on the SoMoF 3DPW validation set
in both VIM and MPJPE. We add each component one-by-one to show their effect
on the final performance. The results indicate that each component benefits
the performance of the model.

Learned Embeddings. Learned joint embeddings allow our method to disambiguate
joint types, just as positional embedding in time-series transformer models
denote input order. The model can use these embeddings to learn relationships
between types joints to better predict human motion dynamics.

Data Augmentation. Our data augmentation methods include random first-frame
selection, sequence flipping, and person permutation. These methods produce a
wide variety of plausible scenes from a small dataset, allowing our model to
generalize well to unseen examples without additional training data. Data
augmentation gives a significant improvement in performance without
architectural changes.

Grid Positioning. We use an embedding to denote the position of all people
within a scene, allowing our model to use global location information. In a
multi-person setting, the location of people in a scene is useful for
predicting social interactions. The attention mechanism of the transformer
exploits the identity and global position of each person to make an effective
multi-person prediction.

###  5.4 Interactions between multiple people

We can validate that SoMoFormer models the interactions between multiple
people by observing the relationship between two people’s distance and the
average attention score imputed between their joint pairs by SoMoFormer.
Figure 4 shows a clear relationship between two people’s distance and the
attention score, providing clear evidence that our model considers
interactions between people in making a prediction.

###  5.5 Visualized Examples

We subjectively evaluate the performance of our model in Figure 1 by
visualizing predicted motion sequences on examples from the SoMoF 3DPW test
set. Comparing the real and predicted sequences, we observe that our model’s
predictions closely match the ground-truth even in a multi-person motion
scenario with two people walking side-by-side. These results validate the
performance of our model.

###  5.6 Attention Weights Visualization

Figure 5: A visualization of attention scores between joint types at different
layers of the model. Each row represents the attention scores from a single
joint as inferred by SoMoFormer. A brighter color indicates a stronger
attention score from the row joint to the column joint.

To further understand our method’s ability to model interactions between
joints, we visualize the attention weights of our model, as well as predicted
attention of various joints in different motion scenarios. In Figure 5 we
visualize the average predicted attention weights across the entire SoMoF
validation set. Figure 5 shows that different layers attend to different input
joints when making predictions, and these attention weights correspond to
realistic human body dynamics. For example, in the Layer 2 attention, joints
on the right side of the body attend strongly to the right hip joint, and
vice-versa. Layers 4 and 5 show strong and symmetrical attention between leg
joints. Interestingly, we note that the model learns to treat joints from the
left and right sides of the body in similar ways, based on nearly identical
attention scores, despite being given no explicit representation of a human
skeleton. Overall, these results show that SoMoFormer leverages human body
dynamics between joints to make accurate motion predictions, confirming the
effectiveness of our joint-based approach.

Figure 6: Qualitative results on the SoMoF dataset. We visualized self-
attention scores from a joint to all other joints, where a brighter color
indicates a stronger attention score. We observe that SoMoFormer learns a rich
representation of human body motion between joints.

Inspired by [16], in Figure 6 we analyze the attention scores for several
motion examples by visualizing the scores between a joint and all other
joints, where a brighter color indicates a stronger attention score. This
visualization shows that (1) Joints tend to have higher attention to
physically closer joints, and (2) In moving scenarios, joints tend to have
higher attention scores to other joints that are most relevant to the motion.
For example, we observe in the first standing scenario, the left foot joint
attends strongly to other leg joints and more weakly to arm joints. However,
in the second walking scenario, the left foot joints attend more strongly to
the neck joint, which defines overall motion, and gives almost no attention to
the arm joints which are not as relevant to the prediction. Figure 6 provides
further evidence that our method learns a rich representation of human body
motion. Further analysis would be needed to fully explore the relationships
between joints in the model predictions.

##  6 Conclusion

In this paper, we proposed SoMoFormer, a novel framework for multi-person pose
forecasting. Using attention over joint trajectories, our method predicts an
entire future pose sequence for multiple people while modeling complex
interactions among body joints. Experiment results show state-of-the-art
performance on the SoMoF benchmark, as well as the CMU-Mocap and MuPoTS-3D
datasets.

Our model can be used in a range of applications, from autonomous vehicle
technology to realistic human motion for skeletal meshes. SoMoFormer can
easily generalize to many people in a scene, and its efficiency in generating
an entire human pose trajectory at once allows it to be used in low latency
applications such as autonomous vehicles and gaming for realistic human
motion. A detailed analysis reveals that SoMoFormer learns rich
representations of human motion by modeling complex relationships between
joints. We hope to further explore the semantic meaning and downstream
applicability of these representations.

## References

  * [1] Adeli, V., Adeli, E., Reid, I.D., Niebles, J.C., Rezatofighi, H.: Socially and contextually aware human motion and pose forecasting. IEEE Robotics and Automation Letters 5, 6033–6040 (2020)
  * [2] Adeli, V., Ehsanpour, M., Reid, I., Niebles, J.C., Savarese, S., Adeli, E., Rezatofighi, H.: Tripod: Human trajectory and pose dynamics forecasting in the wild. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13390–13400 (2021)
  * [3] Adeli-Mosabbeb, E., Fathy, M., Zargari, F.: Model-based human gait tracking, 3d reconstruction and recognition in uncalibrated monocular video. The Imaging Science Journal 60(1), 9–28 (2012)
  * [4] Brand, M., Hertzmann, A.: Style machines. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques. pp. 183–192 (2000)
  * [5] Cai, Y., Huang, L., Wang, Y., Cham, T.J., Cai, J., Yuan, J., Liu, J., Yang, X., Zhu, Y., Shen, X., et al.: Learning progressive joint propagation for human motion prediction. In: European Conference on Computer Vision. pp. 226–242. Springer (2020)
  * [6] kuang Chiu, H., Adeli, E., Wang, B., Huang, D.A., Niebles, J.C.: Action-agnostic human pose forecasting. 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) pp. 1423–1432 (2019)
  * [7] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [8] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. ICLR (2021)
  * [9] Giuliari, F., Hasan, I., Cristani, M., Galasso, F.: Transformer networks for trajectory forecasting. 2020 25th International Conference on Pattern Recognition (ICPR) pp. 10335–10342 (2021)
  * [10] Gong, H., Sim, J., Likhachev, M., Shi, J.: Multi-hypothesis motion planning for visual object tracking. 2011 International Conference on Computer Vision pp. 619–626 (2011)
  * [11] Huang, Y., Bi, H., Li, Z., Mao, T., Wang, Z.: Stgat: Modeling spatial-temporal interactions for human trajectory prediction. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6272–6281 (2019)
  * [12] Koppula, H.S., Saxena, A.: Anticipating human activities for reactive robotic response. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems pp. 2071–2071 (2013)
  * [13] Kosaraju, V., Sadeghian, A., Martín-Martín, R., Reid, I.D., Rezatofighi, S.H., Savarese, S.: Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. In: NeurIPS (2019)
  * [14] Levine, S., Wang, J.M., Haraux, A., Popovic, Z., Koltun, V.: Continuous character control with low-dimensional embeddings. ACM Transactions on Graphics (TOG) 31, 1 – 10 (2012)
  * [15] Li, L.L., Yang, B., Liang, M., Zeng, W., Ren, M., Segal, S., Urtasun, R.: End-to-end contextual perception and prediction with interaction transformer. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) pp. 5784–5791 (2020)
  * [16] Lin, K., Wang, L., Liu, Z.: End-to-end human pose and mesh reconstruction with transformers. In: CVPR (2021)
  * [17] Liu, Y., Zhang, J., Fang, L., Jiang, Q., Zhou, B.: Multimodal motion prediction with stacked transformers. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 7573–7582 (2021)
  * [18] Mahmood, N., Ghorbani, N., Troje, N.F., Pons-Moll, G., Black, M.J.: AMASS: Archive of motion capture as surface shapes. In: International Conference on Computer Vision. pp. 5442–5451 (Oct 2019)
  * [19] Mao, W., Liu, M., Salzmann, M.: History repeats itself: Human motion prediction via motion attention. In: ECCV (2020)
  * [20] Mao, W., Liu, M., Salzmann, M., Li, H.: Learning trajectory dependencies for human motion prediction. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 9488–9496 (2019)
  * [21] Mao, W., Liu, M., Salzmann, M., Li, H.: Multi-level motion attention for human motion prediction. Int. J. Comput. Vis. 129, 2513–2535 (2021)
  * [22] von Marcard, T., Henschel, R., Black, M., Rosenhahn, B., Pons-Moll, G.: Recovering accurate 3d human pose in the wild using imus and a moving camera. In: European Conference on Computer Vision (ECCV) (sep 2018)
  * [23] Mart’inez-Gonz’alez, A., Villamizar, M., Odobez, J.M.: Pose transformers (potr): Human motion prediction with non-autoregressive transformers. 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) pp. 2276–2284 (2021)
  * [24] Mehta, D., Sotnychenko, O., Mueller, F., Xu, W., Sridhar, S., Pons-Moll, G., Theobalt, C.: Single-shot multi-person 3d pose estimation from monocular rgb. In: 2018 International Conference on 3D Vision (3DV). pp. 120–130. IEEE (2018)
  * [25] Parsaeifard, B., Saadatnejad, S., Liu, Y., Mordan, T., Alahi, A.: Learning decoupled representations for human pose forecasting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2294–2303 (2021)
  * [26] Pavlovic, V., Rehg, J.M., MacCormick, J.: Learning switching linear models of human motion. Advances in neural information processing systems 13 (2000)
  * [27] Sadeghian, A., Kosaraju, V., Sadeghian, A., Hirose, N., Savarese, S.: Sophie: An attentive gan for predicting paths compliant to social and physical constraints. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 1349–1358 (2019)
  * [28] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [29] Wang, C., Wang, Y., Huang, Z., Chen, Z.: Simple baseline for single human motion forecasting. 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) pp. 2260–2265 (2021)
  * [30] Wang, J., Xu, H., Narasimhan, M., Wang, X.: Multi-person 3d motion prediction with multi-range transformers. Advances in Neural Information Processing Systems 34 (2021)
  * [31] Wu, D., Shao, L.: Leveraging hierarchical parametric networks for skeletal joints based action segmentation and recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 724–731 (2014)
  * [32] Yu, C., Ma, X., Ren, J., Zhao, H., Yi, S.: Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In: ECCV (2020)