††thanks: contributed equally††thanks: contributed equally††thanks:
<EMAIL_ADDRESS>
# Quantum Liquid: From the perspective of Surface Tension

Sri Satwika Adusumalli  Department of Physics, IIT Roorkee, Roorkee, 247667,
Uttarakhand, India  Kathaa Senapati  Dipartimento di Fisica e Astronomia
“Galileo Galilei”, Università degli Studi di Padova, Via Marzolo, 8 - 35131
Padova, Italy  Shivam Singh  Ayan Khan  Department of Physics, School of
Engineering and Applied Sciences, Bennett University, Greater Noida,
UP-201310, India

###### Abstract

We analyze the surface tension in ultra-cold atomic gases in quasi one
dimensional and one dimensional geometry. In recent years, experimental
observations have confirmed the “clustering of atoms” to form droplets in
ultra-cold atomic gases and the emergence of this new phase is attributed to
the beyond mean-field interaction. However, two decades earlier, liquid
formation was predicted due to the competition of two-body and three-body
interactions. Here, we review both the propositions and comment on the role of
beyond mean-field and three-body interaction in liquid formation by
calculating the surface tension.

##  I Introduction

The physics at ultra-low temperature is always a matter of extreme interest.
In this direction a new impetus has been provided through the experimental
observations of liquid-like phase in ultra-cold gases Kadau  _et al._ (2016);
Li  _et al._ (2017); Cabrera  _et al._ (2018); Chomaz  _et al._ (2019);
Böttcher  _et al._ (2019); Tanzi  _et al._ (2019); Donner (2019). The
experimental observation is treated as verification of the suggested
theoretical framework offered recently Petrov (2015); Petrov and Astrakharchik
(2016). In the current theoretical model, the beyond mean-field (BMF)
contribution (proposed by Lee-Huan-Yang Lee  _et al._ (1957)) is considered as
the catalyst in the stabilization mechanism which in effect supports the
formation of the liquid-like droplets (can be called as dropleton). One of the
prominent signatures of the formation of liquid-like states is the homogeneous
spatial density Ferrier-Barbut (2019).

However, one must note that, about two decades ago, a similar state of matter
was predicted based on the Efimov effect which results in creation of three-
body bound states Bulgac (2002); Bedaque  _et al._ (2003); Bulgac (2020). The
competition between the two-body mean-field interaction (cubic non-linearity)
and three-body interaction (quintic non-linearity) is at the heart of this
theory. The liquid formation was substantiated through the calculation of non-
zero surface tension.

In recent years there have been some suggestions that the flat-top density
distribution arising from the competition between cubic-quintic non-linearity,
actually points to flat-top solitons (FTS) or platicons which are different
from the droplets in its dynamical nature Lobanov  _et al._ (2021); Alotaibi
_et al._ (2023). In this context, we plan to investigate the ambiguity
associated with dropleton and platicon by analyzing the non-linear Schrodinger
equation with LHY correction and three-body interaction (as suggested in
Ref.Bulgac (2002)) in one dimensional (1D) and quasi 1D (Q1D) systems.

The theoretical discussions on droplet formation in 1D Bose gas or in a Q1D
binary Bose-Einstein condensate (BEC) revolves around the nature of the BMF
interaction. To be more precise, in the 1D system the BMF interaction
manifests via quadratic non-linearity Petrov and Astrakharchik (2016);
Astrakharchik and Malomed (2018) while it is quartic non-linearity in Q1D
Debnath and Khan (2021). Our primary objective is to study the competition
between two-body mean-field interaction, BMF and three-body interaction in
liquid formation. For this purpose, we will rely on the calculation of surface
tension.

We have arranged our results in the following way: in Sec. II we summarize the
recent theoretical model to describe Q1D and 1D systems after taking into
account the LHY contribution. Later, we note down the equation of motion in
Q1D and 1D with three-body interaction, which we describe as cubic-quartic-
quintic non-linear Schrödinger equation (CQQNLSE) and quadratic-cubic-quintic
non-linear Schrödinger equqaiton (QCQNLSE) respectively. In the next section
i.e., in Sec.III we explicate the analytical solutions and calculate the
surface tension. Our findings relate to both the dimensions and we identify
the role of interaction as well as dimensionality in liquid formation. We draw
our conclusion in Sec.IV.

##  II Theoretical Models

In this section we plan to summarize the existing theories for both Q1D and
1D. The theory of Q1D is derived from the three dimensional model Petrov
(2015) with suitable dimension reduction technique (which we will touch upon
here) Debnath and Khan (2021) and the 1D theory is provided in Ref. Petrov and
Astrakharchik (2016).

### Quasi-One-Dimensional System

Figure 1: Schematic representation of the competition between inter-species
and intra-species interactions in atomic two component BEC and associated
quantum states. The red and blue circles represent two different atomic
species.

In two-component BEC or binary BEC (the two components can be two different
species of atoms Pu and Bigelow (1998) or two-different hyper-fine states of
same species Myatt  _et al._ (1997)) due to the competition between inter-
species and intra-species interactions one can observe distinct mixing
dynamics (as schematically described in Fig. 1). The intra-species coupling is
described via $g_{11}$ and $g_{22}$ while the inter-species interaction is
denoted as $g_{12}$. If $g_{11},g_{22}$ and $g_{12}$ all are repulsive then we
can look forward to transition between miscible and the immiscible phase. The
condensate will be in miscible phase when
$-\sqrt{g_{11}g_{22}}<g_{12}<\sqrt{g_{11}<g_{22}}$, and will be in immiscible
phase when $g_{12}>\sqrt{g_{11}g_{22}}$ while the condensate will collapse for
$-\sqrt{g_{11}g_{22}}>g_{12}$.

In a uniform Bose mixture, having two components of masses $m_{1}$ and $m_{2}$
Ancilotto  _et al._ (2018), the coupling constant of intra-
species($g_{11}=4\pi a_{11}\hbar^{2}/m_{1},\hskip 5.69054ptg_{22}=4\pi
a_{22}\hbar^{2}/m_{2}$) and inter-species($g_{12}=2\pi a_{12}\hbar^{2}/m_{r}$)
interactions, where $m_{r}$ is the reduced mass ($m_{1}m_{2}/(m_{1}+m_{2}$)
and, $a_{11}$ and $a_{22}$ are the intra-species interaction, both are
positive, meanwhile inter-species $a_{12}$ is negative. If number densities
are $n_{1}$ and $n_{2}$ then there normalized form can be represented
respectively as : $\int_{V}n_{1}dr=N_{1}$ and $\int_{V}n_{2}dr=N_{2}$. Here,
$N=N_{1}+N_{2}$ represents the total number of bosons in the mixture along
with the total number density, $n=n_{1}+n_{2}$. Therefore, energy functional
can be written as Cappellaro  _et al._ (2018),

$\displaystyle E$ $\displaystyle=$
$\displaystyle\sum_{j=1,2}\Big{[}\frac{\hbar^{2}}{2m}|\nabla\psi_{j}|^{2}+V_{ex}(r)|\psi_{j}|^{2}+\frac{1}{2}g_{jj}|\psi_{j}|^{4}\Big{]}$
(1)
$\displaystyle+g_{12}|\psi_{1}|^{2}|\psi_{2}|^{2}+E_{BMF}(\psi_{1},\psi_{2}).$

Here, $E_{BMF}$ stands for the beyond mean-field interaction energy. As
described in Fig. 1, near the line of collapse where the effective interaction
$g_{12}+\sqrt{g_{11}g_{22}}\rightarrow 0$, the relatively weak BMF interaction
plays important role in stabilizing the system. For the case when we had gone
close to the MF instability boundary i.e $\delta g<0$, we need to avoid the
imaginary contribution for which we approximate that the instability is very
weak i.e $|\delta g|<<g$ and hence we assume $|\delta g|\approx 0$.

Now, considering that density ratio will be $n_{1}/n_{2}=\sqrt{g_{22}/g_{11}}$
= $\eta$, via two components have indistinguishable spatial mode
$n_{i=1,2}=|\psi|^{2}$. Therefore, $g$ = $\sqrt{g_{11}g_{22}}$ and $\delta g$
= $g+\sqrt{g_{11}g_{22}}$. The energy functional in Eq.(1) can now be
rewritten as,

$\displaystyle E$ $\displaystyle=$
$\displaystyle\frac{\hbar^{2}}{2m}|\nabla\psi|^{2}+V_{trap}|\psi|^{2}+\delta
g\frac{\eta}{(\eta+1)^{2}}|\psi|^{4}$ (2)
$\displaystyle+\frac{8m^{3/2}}{15\pi^{2}\hbar^{3}}(\eta
g_{11})^{5/2}|\psi|^{5}$

The corresponding effective one component equation of motion can be noted as
Cheiney  _et al._ (2018),

$i\hbar\frac{\partial\psi}{\partial
t}=\Big{[}-\frac{\hbar^{2}}{2m}\nabla^{2}+V_{trap}+U|\psi|^{2}+U^{{}^{\prime}}|\psi|^{3}\Big{]}\psi,$
(3)

where, $U$ = $2\delta g\frac{\eta}{(1+\eta)^{2}}n$ and $U^{{}^{\prime}}$ =
$\frac{4m^{3/2}}{3\pi^{2}\hbar^{3}}(ng_{11})^{5/2}n^{3/2}$. Here, it is
important to note that the above equation contains two types of non-linearity:
one is cubic non-linearity, which is responsible for the mean field (MF)
interaction and another is quartic non-linearity which is justification of
beyond mean field (BMF) interaction.

In Eq.(3), the external potential $V_{trap}$ has two components: one in the
transverse direction, defined as
$V_{T}(y,z)=\frac{1}{2}m\omega^{2}_{\perp}(y^{2}+z^{2})$ and the other in the
longitudinal direction, noted as $V_{L}(x)=\frac{1}{2}m\omega_{o}^{2}x^{2}$
where $\omega_{\perp}$ and $\omega_{o}$ is the transverse and longitudinal
trap frequency respectively Khaykovich  _et al._ (2002). One can now
conveniently reduce Eq.(3) from $3+1$ dimension to $1+1$ dimension by setting
$\omega_{\perp}$ as ten times the $\omega_{o}$. Hence, by tuning the trapping
frequency it is possible to create a cigar shaped BEC in quasi one dimension.
For a trapped gas the characteristic length scale is defined as
$a_{\perp}=\sqrt{\frac{\hbar}{m\omega_{\perp}}}$, however, in quasi one
dimension it would be, $a_{x}/a_{\perp}\approx\sqrt{10}$. Here, $a_{x}$ will
be $a_{x}=\sqrt{\frac{\hbar}{m\omega_{o}}}$. In order to reduce the dimension
of Eq.(3) in an effective one dimensional (Q1D) equation we consider an ansatz
of the following form Khan and Panigrahi (2013),

$\psi(r,t)=\frac{1}{\sqrt{2\pi
a_{B}a_{\perp}}}\psi\Big{(}\frac{x}{a_{\perp}},\omega_{\perp}t\Big{)}e^{\Big{(}-i\omega_{\perp}t-\frac{y^{2}+z^{2}}{a_{\perp}^{2}}\Big{)}}$
(4)

A systematic and careful calculation then leads to the Q1D CQNLSE (here, we
have ignored the details for brevity),

$\displaystyle i\frac{\partial\psi(r,t)}{\partial t}$ $\displaystyle=$
$\displaystyle\left[-\frac{1}{2}\frac{\partial^{2}\psi(x,t)}{\partial
x^{2}}+\frac{1}{2}Kx^{2}+G_{1}|\psi(x,t)|^{2}\right.$ (5)
$\displaystyle\left.+G_{4}|\psi(x,t)|^{3}\right]\psi(x,t),$

where $G_{1}=\frac{\delta gnm}{4\pi a_{B}\hbar^{2}}$,
$G_{4}=\frac{\sqrt{2}}{3}\frac{m^{3/2}n^{3/2}g_{sym}^{5/2}}{\pi^{2}a_{B}^{3/2}a_{\perp}\hbar^{5}}$,
$g_{sym}=g_{11}=g_{22}$ and $K={\omega_{o}^{2}}/{\omega_{\perp}^{2}}$. In a
more generic manner we can write the equation as,

$\displaystyle i\frac{\partial\psi(r,t)}{\partial t}$ $\displaystyle=$
$\displaystyle\left[-\frac{1}{2}\frac{\partial^{2}\psi(x,t)}{\partial
x^{2}}+V(x)+g_{1}|\psi(x,t)|^{2}\right.$ (6)
$\displaystyle\left.+g_{4}|\psi(x,t)|^{3}\right]\psi(x,t),$

Here, $g_{1}$ and $g_{4}$ stands for the MF and BMF interaction strength while
$V(x)$ is the external potential.

### One Dimensional System

Contrary to the Q1D picture (derived from three dimensional setup), in 1D Bose
gas the BMF contribution manifests as quadratic non-linearity. Further, one
can also note that the nature of MF and BMF interaction is reversed in 1D,
i.e., MF interaction is repulsive and BMF interaction is attractive.

In 1D system, we define, $\delta g$ = $g_{12}+\sqrt{g_{11}g_{22}}$ which is in
the vicinity of unstable region which implies $0<\delta
g<\sqrt{g_{11}g_{22}}$. Therefore, to explain the one dimension GP equation
including the beyond mean field (BMF) correction, a small variation between
the inter-component attraction strength $(g_{12<0})$ and intra-component self-
repulsion, $\delta g=g+g_{12}>0$ along with $g<0$, can be expressed as Debnath
_et al._ (2022),

$i\hbar\frac{\partial\psi}{\partial
t}=-\frac{\hbar^{2}}{2m}\frac{\partial^{2}\psi}{\partial
x^{2}}+V(x)\psi+\delta
g|\psi|^{2}\psi-\frac{\sqrt{2m}}{\pi\hbar}g^{3/2}|\psi|\psi,$ (7)

where $V(x)$ is the external potential and $m$ is the atomic mass. For further
analysis we write the equation motion in a more generic form such that,

$i\frac{\partial\psi}{\partial t}=-\frac{1}{2}\frac{\partial^{2}\psi}{\partial
x^{2}}+V(x)\psi+g_{3}|\psi|^{2}\psi+g_{2}|\psi|\psi.$ (8)

Here, $g_{3}$ and $g_{2}$ denote the strength of the MF and BMF interactions
respectively.

### Inclusion of Three-Body Interaction

In the earlier subsections, we have discussed the inclusion of BMF
contribution in the GP equation at lower dimensions. However, exactly two
decades ago, we found the first suggestion of droplet formation due to the
competition between two-body MF interaction and three-body interaction Bulgac
(2002); Bedaque  _et al._ (2003); Bulgac (2020). Here, we plan to explicate
the formalism introduced in Ref. Bulgac (2002).

It is well known that, for a dilute Bose system if two-body scattering length
is very large i.e., $a>>r_{0}$, where $r_{0}$ is the range of the potential,
we start observing three-body bound state formation. This is commonly
described as the Efimov effect Efimov (1973). In this regime there are two
independent dimensionless parameters $\rho a^{3}$ and $\rho r_{0}^{3}$. The
three-body bound states appear irrespective of whether the two-body scattering
length is positive or negative. It is further noted that, if the three-body
zero-energy scattering amplitude defined as $g_{3}$ then the contribution of
the three-particle collisions to the ground state energy density of a dilute
Bose gas can be expressed as,
$\mathcal{E}_{3}(\rho)=\frac{1}{6}g_{3}\rho^{3}$. In this situation, the
ground state energy for N bosons in an ensemble can be noted as,

$E(N)=\int
d^{3}r\Big{[}\frac{\hbar^{2}}{2m}|\nabla\psi(r)|^{2}+\frac{1}{2}g_{1}\rho(r)^{2}+\frac{1}{6}g_{3}\rho(r)^{3}\Big{]}$
(9)

Here, $\rho(r)=|\psi(r)|^{2}$ is the number density. Also
$g_{1}={4\pi\hbar^{2}a}/{m}$ ($a$ = two particle scattering length and $m$ =
particle mass).

Following the usual prescription of energy minimization and applying the
dimensional reduction method expressed in earlier, we can write the equation
of motion as,

$i\hbar\frac{\partial\psi}{\partial
t}=-\frac{\hbar^{2}}{2m}\frac{\partial^{2}\psi}{\partial
x^{2}}+\frac{g_{1}}{2}|\psi|^{2}\psi+\frac{g_{3}}{6}|\psi|^{4}\psi$ (10)

However, accounting the result from Eq. 6 and Eq. 8 and incorporating the
three-body interaction in these equations the effective cubic-quartic-quintic
NLSE (CQQNLSE) and quadratin-cubic-quintic NLSE (QCQNLSE) will read:

$\displaystyle i\frac{\partial\psi}{\partial t}$ $\displaystyle=$
$\displaystyle-\frac{1}{2}\frac{\partial^{2}\psi}{\partial
x^{2}}+g_{1}|\psi|^{2}\psi+g_{4}|\psi|^{3}\psi$ (11)
$\displaystyle+g_{3}|\psi|^{4}\psi+V(x)\psi$ $\displaystyle
i\frac{\partial\psi}{\partial t}$ $\displaystyle=$
$\displaystyle-\frac{1}{2}\frac{\partial^{2}\psi}{\partial
x^{2}}+g_{1}|\psi|^{2}\psi+g_{2}|\psi|\psi$ (12)
$\displaystyle+g_{3}|\psi|^{4}\psi+V(x)\psi$

where, $g_{1}$ and $g_{3}$ are the two-body and three body interaction
strength respectively. $g_{2}(g_{4})$ describes the BMF interaction in 1D
(Q1D) geometry while $V(x)$ stands for external potential. In the next section
we will propose the possible analytical solution. Further, we will comment on
the potential structures which are essential to stabilize the solutions. Also
in the above equations, we have not made any specific comment on the nature of
the interactions which we plan to explicate in the next section.

##  III Analytical solutions

In the preceding section we have discussed the equation of motion in different
geometries and explicated the intriguing nature of the BMF interaction while
moving from Q1D to 1D system. One can find a more detailed discussion on this
aspect in Ref. Debnath  _et al._ (2022). Here, we plan to examine the
existence of cnoidal solutions for the above mentioned equations. For this
purpose we follow the similar flow description as earlier and discuss Q1D
geometry first and then we elaborate on the 1D system.

### Analytical Solution in Q1D

Now, we are interested in finding out the solution of Eq.(11) by considering
trapping potential in a cnoidal form Abramowitz  _et al._ (1988); Debnath  _et
al._ (2022). Further, we assume the temporal evolution is sinusoidal such
that, $\psi(x,t)=\phi(x)e^{-i\alpha t}$. Here, $\alpha$ can be interpreted as
chemical potential. We also use $z=\zeta x$ so that $z$ is dimensionless and
$\zeta$ can be interpreted as inverse of the coherence length. The cnoidal
potential is defined as, $V(z)=V_{0}cn^{4}(z,m)$. The form of the potential
plays an important role in stabilizing the solution. Hence, the modified
CQQNLSE will read,

$\displaystyle\zeta^{2}\frac{\partial^{2}\phi}{\partial x^{2}}-\phi(z)[\alpha-
g_{1}|\phi(z)|^{2}-g_{4}|\phi(z)|^{3}$ $\displaystyle-
g_{3}|\phi(z)|^{4}-V_{0}cn^{4}(z,m)]=0.$ (13)

To find an analytical solution for Eq.(13) we consider an ansatz of the form
$\phi(z)=A+Bcn(z,m)$, $m$ being the moduli parameter.

So, by applying the ansatz in Eq.(13), we derive the set of consistency
conditions such that,

$\displaystyle-A\alpha+A^{3}g_{1}+A^{4}g_{4}+A^{5}g_{3}=0$ (14)
$\displaystyle-B(\alpha+\zeta^{2}-2m\zeta^{2}-3A^{2}g_{1}-4A^{3}g_{4}-5A^{4}g_{3})=0$
(15) $\displaystyle AB^{2}(3g_{1}+2A(3g_{4}+5Ag_{3}))=0$ (16)
$\displaystyle-2Bm\zeta^{2}+B^{3}g_{1}+4AB^{3}g_{4}+10A^{2}B^{3}g_{3}=0$ (17)
$\displaystyle B^{4}g_{4}+A(5B^{4}g_{3}+V_{0})=0$ (18) $\displaystyle
B(B^{4}g_{3}+V_{0})=0$ (19)

Solving and rearranging the set of equations Eq.(14) to Eq.(19), we are able
to find solution parameter ($A$ and $B$) in terms of the equation parameters
($V_{0}$, $g_{3}$ and $g_{4}$).

${B=\Big{(}-\frac{V_{0}}{g_{3}}\Big{)}^{\frac{1}{4}}},\hskip
14.22636pt{A=-\frac{g_{4}}{4g_{3}}}.$ (20)

Further, we obtain a set of constrained condition such as,

$\displaystyle g_{1}$ $\displaystyle=$
$\displaystyle\frac{7g_{4}^{2}}{24g_{3}}$ (21) $\displaystyle\alpha$
$\displaystyle=$ $\displaystyle\frac{5g_{4}^{2}}{768g_{3}^{3}}$ (22)
$\displaystyle\zeta$ $\displaystyle=$
$\displaystyle\sqrt{\frac{g_{4}^{2}}{12g_{3}}\Big{[}\frac{g_{4}^{2}}{16g_{3}}-i\sqrt{\frac{V_{0}}{g_{3}}}\Big{]}}$
(23)

The constrained conditions allow us to examine the competition between the
three interaction strengths. From Eq.(20) and Eq.(23) we realize that the
three body interaction strength ($g_{3}$) or the strength of the external
potential ($V_{0}$) must be negative such that $B>0$. This will also ensure
that the inverse of coherence length is real. Moreover, later we will see that
the same condition is also mandatory to obtain real value of surface tension.
From the above set of equations, we can also conclude that the nature of BMF
interaction is redundant in the current context which implies that it can be
attractive or repulsive in nature without making any significant change in the
obtained results. However, Eq.(21) implies that the mean-field interaction is
attractive and Eq.(22) tells us that $\alpha$, which can be treated as scaled
chemical potential, is negative.

From a deeper examination of the interplay between the interactions, we
realize that MF interaction has parabolic dependence on BMF interaction as
depicted in Fig. 2. Interestingly, if BMF interaction is zero then by Eq.(21)
the MF interaction is also nonexistent, which leads to infinite coherence
length (see Eq.(23)) and unacceptable solution. Hence, a small contribution of
BMF interaction is essential for a physically meaningful solution. Fig. 3 and
Eq.(21) reveals that the MF and three body interactions are inversely
proportional which resonates with our intuitions. Also, it is worth noting
that $g_{4}\propto\sqrt{g_{1}g_{3}}$.

Figure 2: Variation of two body interaction as a function of BMF term (for
$g_{3}=2$). The figure clearly suggests a parabolic dependence between these
two interactions. The scale and the parameter value is in an arbitrary unit.
Figure 3: Variation of two-body interaction with three-body interaction
strength for a fixed BMF interaction. The scale and the parameter value is in
arbitrary units.

The final solution can be noted as,

${\phi(z)=-\frac{g_{4}}{4g_{3}}+\Big{(}-\frac{V_{0}}{g_{3}}\Big{)}^{\frac{1}{4}}cn(z,m)}.$
(24)

We have depicted the density distribution obtained from Eq.(24) for $m=0$ and
$m=1$ in Fig. 4 and 5 respectively. The density variation is scaled by the
density at the centre of trap, i.e.,
$|\phi_{0}|^{2}=|\phi(z=0)|^{2}=\left(\frac{g_{4}}{4g_{3}}\left(\left(\frac{|V_{0}|}{g_{3}}\right)^{1/4}\frac{4g_{3}}{g_{4}}-1\right)\right)^{2}$.
For $m=0$ we obtain a periodic but binary amplitude distribution. The
localized solution corresponding to $m=1$ depicts a $w$-soliton like structure
with finite background density Debnath and Khan (2020).

Figure 4: (Colour Online) The figure depicts the density distribution (blue
solid line) and the variation of the external potential (red dotted line). The
density is scaled by $|\phi_{0}|^{2}=|\phi(z=0)|^{2}$. For better visibility
it is further scaled by the peak value. $V(z)$ is in the units of $V_{0}$.
Here, $m=0$; $g_{4}=2$; $g_{3}=1$; $V_{0}=-1$. The parameter values are in
arbitrary units. Figure 5: (Colour Online) Density distribution (blue solid
line) and spatial variation of the external potential (red dotted line) is
described here (scaled by $|\phi_{0}|^{2}$). The figure is prepared for $m=1$;
$g_{4}=2$; $g_{3}=1$; $V_{0}=-1$. The parameter values are in arbitrary units.

### Analytical Solution in 1D

We now repeat the similar strategy to solve QCQNLSE as described in Eq.(12).
We assume, $\psi(x,t)=\phi(z)e^{-i\alpha t}$ with ansatz solution as
$\phi(z)=Bcn(z,m)$. As described earlier, $z=\eta x$, and $\eta$ is the
inverse of coherence length. We realize that to stabilize the system we
require a super-lattice type potential such that,
$V(z)=V_{0}cn(z,m)+V_{1}cn^{4}(z,m)$ Debnath  _et al._ (2022).

Now Eq.(12) in $z$ variable can be written as,

$\displaystyle\eta^{2}\frac{\partial^{2}\phi}{\partial z^{2}}+\phi(z)[\alpha-
g_{1}|\phi(z)|^{2}-g_{2}|\phi(z)|-g_{3}|\phi(z)|^{4}$ $\displaystyle-
V_{0}cn(z,m)-V_{1}cn^{4}(z,m)]=0$ (25)

Inserting the ansatz solution we obtain a set of consistency equations which
are noted as,

$\displaystyle B\alpha-B\eta^{2}+2Bm\eta^{2}$ $\displaystyle=$ $\displaystyle
0$ (26) $\displaystyle-B^{2}g_{2}-BV_{0}$ $\displaystyle=$ $\displaystyle 0$
(27) $\displaystyle-B^{3}g_{1}-2Bm\eta^{2}$ $\displaystyle=$ $\displaystyle 0$
(28) $\displaystyle-B^{5}g_{3}-BV_{1}$ $\displaystyle=$ $\displaystyle 0$ (29)

An appropriate rearrangement will lead to the solution of the Eqs. (26), (27),
(28) and (29). We yield a relationship between the equation parameters and
solution parameters in the following form,

${B=-\frac{V_{0}}{g_{2}}},\hskip
14.22636pt{g_{1}=-\frac{2m\eta^{2}}{V_{0}^{2}}g_{2}^{2}}.$ (30)

Thus, the mean-field interaction is proportional to the square of the BMF
interaction. Further, we obtain a constrained relationship between the
interactions ($g_{2}$ and $g_{3}$) and potential strengths ($V_{0}$ and
$V_{1}$) such that $\frac{g_{2}^{4}}{g_{3}}=-\frac{V_{0}^{4}}{V_{1}}$. The
chemical potential and the inverse of coherence length reads:

$\displaystyle\alpha$ $\displaystyle=$
$\displaystyle-\frac{V_{0}^{2}(1-2m)}{2m}\Big{(}\frac{g_{1}}{g_{2}^{2}}\Big{)},$
(31) $\displaystyle\eta$ $\displaystyle=$
$\displaystyle\sqrt{\frac{\alpha}{1-2m}}.$ (32)

Figure 6: (Colour Online) Variation of density (blue solid line) and
description of potential landscape (red dotted line) is illustrated here (with
moduli parameter, $m=1$). The density is scaled by the centre of the trap
density, i.e., $|\phi_{0}|^{2}=|\phi(z=0)|^{2}=(-V_{0}/g_{2})^{2}$. The
potential is in units of $V_{0}$. All the parameters are in arbitrary units
such that $g_{2}=1$; $V_{0}=-1$; $V_{1}=1$.

Therefore, the final solution turns out as,

${\phi(z)=-\frac{V_{0}}{g_{2}}cn(z,m)}$ (33)

A closer inspection of Eqs.(31) and (32) allows us to conclude that at $m=0$,
$\eta\rightarrow\infty$ so to keep the dimensionless parameter $z$ finite,
$x\rightarrow 0$. Also, Eq.(33) reveals to us that it is essential to have the
BMF interaction for a physically acceptable solution. In Fig. 6 we depict the
density distribution via blue solid line and external trap variation through
red dotted line. The solution clearly manifests bright soliton like nature
where $\phi(z)\rightarrow 0$ asymptotically. We also note the periodic
solution against the backdrop of quasi periodic potential landscape in Fig. 7.

Figure 7: (Colour Online) The figure describes the periodic variation of the
density mode (blue solid line) in a super-lattice like potential landscape
(red dotted line). The moduli parameter or $m$ is considered as zero here. The
density is scaled by the centre of the trap density, i.e.,
$|\phi_{0}|^{2}=|\phi(z=0)|^{2}=(-V_{0}/g_{2})^{2}$. The other parameters are
in arbitrary units such as $g_{2}=1$; $V_{0}=-1$ and $V_{1}=1$.

### Calculation of Surface Tension

Surface tension is a solely liquid property that allows it to resist an
external force, due to the cohesive nature of its molecules. Hence, in the
context of liquid formation in quantum gases, existence of surface tension can
be regarded as one of the significant pieces of evidence as pointed out in
Ref. Bulgac (2002).

In quantum gases, if the effective scattering length is attractive (say $a<0$)
then energy per particle relation for a uniform condensate(i.e
$E_{2}=\frac{1}{2}g_{2}\rho$ , where $g_{2}=\frac{4\pi\hbar^{2}a}{m}$), is
unbounded and density be will unstable with respect to the density
fluctuations. However, it was suggested that inclusion of three-body
correlation effect changes the situation dramatically and predicted that
stable droplet formation can be possible via surface tension calculation
Bulgac (2002).

If two body scattering length is large enough(i.e $|a|>>r_{0}$, where, $r_{0}$
is the two body interaction radius) then there will be two independent
dimensionless parameter $\rho a^{3}$ and $\rho r_{0}^{3}$. In this situation,
three-body interaction starts playing significant role irrespective of the
nature of two-body interaction (i.e., whether it is attractive or repulsive),
eventually we observe emergence of three-body bound state which is commonly
known as Efimov effect Efimov (1973). In this situation, the surface tension
can be defined as,

$\sigma=\int_{-\infty}^{\infty}dx[\epsilon(x)-\mu_{0}\rho(x)],$ (34)

where $\epsilon(x)$ is the energy density and $\rho(x)$ is the density. Now
applying Eq.(24) and Eq.(33) in Eq.(34) using the energy functional
corresponding to Eq.(11) and Eq.(12) we can obtain the non zero surface
tension for Q1D and 1D respectively,

$\displaystyle\sigma_{Q1D}$ $\displaystyle=$
$\displaystyle\frac{1}{3}\sqrt{-\frac{V_{0}}{g_{3}}},$ (35)
$\displaystyle\sigma_{1D}$ $\displaystyle=$
$\displaystyle\frac{1}{3}\left(\frac{V_{0}}{g_{2}}\right)^{2}.$ (36)

The calculation is carried out after carefully transforming the $x$ variable
to $z$ variable.

Eq.(35) and Eq.(36) possess some common attributes while they are quite
distinctive in nature as well. In both cases we find that the trapping
potential has an important role to play and if $V_{0}=0$ then surface tension
is zero which implies that the atoms can not form clusters to emerge as
liquid-like state. However, one striking difference is, in Q1D three-body
interaction is essential for non-zero surface tension while in 1D the BMF
interaction plays that role. Fig. 8 describes that the surface tension is
proportional to the inverse square root of three body interaction while in
Fig. 9 we observe that the surface tension varies with the inverse square of
BMF interaction. This implies that, in Q1D we should be able to yield quantum
droplets even if we do not take into account BMF interaction which was
precisely pointed out in Ref. Bulgac (2020), nevertheless in 1D system, we see
a role reversal.

If we now look into the chronological perspective, the recent discussions on
droplets are mainly from the perspective of BMF interaction; the earlier
discussion was more in the context of interplay between two-body and three-
body interactions. However, our current discussion suggests that BMF
interaction is irreplaceable in 1D, while in Q1D, three-body interaction can
induce liquid formation. Hence, based on the recent discussions on platicons
(FTS) and dropletons Lobanov  _et al._ (2021); Alotaibi  _et al._ (2023), and
our current observation, motivates us to characterize the Q1D liquid as
platicon and 1D liquid as dropleton.

Figure 8: Variation of surface tension as a function of three body interaction
in Q1D system. Here, $V_{0}$ is considered as $-1$ in arbitrary units. Figure
9: Variation of surface tension with BMF interaction in 1D system. Here,
$V_{0}$ is considered as $-1$ in arbitrary units.

##  IV Conclusion

In recent years, we have observed significant interest in understanding
liquid-like state formation in ultra-cold atomic gases due to the theoretical
developments Petrov (2015); Petrov and Astrakharchik (2016); Ilg  _et al._
(2018) and experimental verification Cabrera  _et al._ (2018); Ferrier-Barbut
_et al._ (2016); Wächtler and Santos (2016). The fundamental theoretical
proposition suggests that the BMF interaction competes with the effective MF
interaction to support the clustering of atoms. However, two decades ago a
similar exotic phase was predicted theoretically by considering the
competition of two-body and three-body interactions Bulgac (2002); Bedaque
_et al._ (2003). In this article, we have tried to review both the
perspectives in Q1D and 1D geometry through a pathelogical model.

We have already noted interesting differences in Q1D and 1D geometry from
mathematical and physical perspective Debnath  _et al._ (2022) when we include
BMF interaction. We have noted that in Q1D, the dynamical equation boils down
to a CQNLSE while in 1D it is QCNLSE. We have now included the three-body
interaction in our calculation and obtained the cnoidal solutions. We know
that the cnoidal functions can boil down to periodic or localized function
based on their moduli parameter. Hence, to exploit this mathematical
manoeuvrability we restrict our solutions to cnoidal functions only. Later, we
calculate the corresponding surface tension associated with each dimension.
Interestingly, we note that an external potential is invariably required for
non-zero surface tension in both the cases. However, in Q1D, surface tension
is independent of BMF interaction and we can have non-zero surface tension due
to the competition between two-body and three-body interactions while in 1D,
it is essential to have BMF interaction whereas three-body interaction appears
redundant.

We hope our current analysis will shed some light in the context of dropleton
and platicon and will also motivate some experiments to study these exotic
phases in lower dimensions.

## Acknowledgement

AK thanks Council of Scientific and Industrial Research (CSIR) Human Resource
Development Group (HRDG) Extramural Research Division (EMR-II), India for the
support provided through the project number 03/1500/23/EMR-II.

## References

  * Kadau  _et al._ (2016) H. Kadau, M. Schmitt, M. Wenzel, C. Wink, T. Maier, I. Ferrier-Barbut, and T. Pfau, Nature 530, 194 (2016).
  * Li  _et al._ (2017) J.-R. Li, J. Lee, W. Huang, S. Burchesky, B. Shteynas, F. Ç. Top, A. O. Jamison, and W. Ketterle, Nature 543, 91 (2017).
  * Cabrera  _et al._ (2018) C. Cabrera, L. Tanzi, J. Sanz, B. Naylor, P. Thomas, P. Cheiney, and L. Tarruell, Science 359, 301 (2018).
  * Chomaz  _et al._ (2019) L. Chomaz, D. Petter, P. Ilzhöfer, G. Natale, A. Trautmann, C. Politi, G. Durastante, R. Van Bijnen, A. Patscheider, M. Sohmen, _et al._ , Physical Review X 9, 021012 (2019).
  * Böttcher  _et al._ (2019) F. Böttcher, J.-N. Schmidt, M. Wenzel, J. Hertkorn, M. Guo, T. Langen, and T. Pfau, Physical Review X 9, 011051 (2019).
  * Tanzi  _et al._ (2019) L. Tanzi, E. Lucioni, F. Famà, J. Catani, A. Fioretti, C. Gabbanini, R. N. Bisset, L. Santos, and G. Modugno, Physical review letters 122, 130405 (2019).
  * Donner (2019) T. Donner, Physics 12, 38 (2019).
  * Petrov (2015) D. S. Petrov, Phys. Rev. Lett. 115, 155302 (2015).
  * Petrov and Astrakharchik (2016) D. S. Petrov and G. E. Astrakharchik, Phys. Rev. Lett. 117, 100401 (2016).
  * Lee  _et al._ (1957) T. D. Lee, K. Huang, and C. N. Yang, Phys. Rev. 106, 1135 (1957).
  * Ferrier-Barbut (2019) I. Ferrier-Barbut, Physics Today 72, 46 (2019).
  * Bulgac (2002) A. Bulgac, Physical review letters 89, 050402 (2002).
  * Bedaque  _et al._ (2003) P. F. Bedaque, A. Bulgac, and G. Rupak, Physical Review A 68, 033606 (2003).
  * Bulgac (2020) A. Bulgac, Physics Today 73, 10 (2020).
  * Lobanov  _et al._ (2021) V. E. Lobanov, A. E. Shitikov, R. R. Galiev, K. N. Min’kov, O. V. Borovkova, and N. M. Kondratiev, Physical Review A 104, 063511 (2021).
  * Alotaibi  _et al._ (2023) M. Alotaibi, L. Al Sakkaf, and U. Al Khawaja, Physics Letters A 487, 129120 (2023).
  * Astrakharchik and Malomed (2018) G. E. Astrakharchik and B. A. Malomed, Phys. Rev. A 98, 013631 (2018).
  * Debnath and Khan (2021) A. Debnath and A. Khan, Annalen der Physik 533, 2000549 (2021).
  * Pu and Bigelow (1998) H. Pu and N. Bigelow, Physical review letters 80, 1130 (1998).
  * Myatt  _et al._ (1997) C. Myatt, E. Burt, R. Ghrist, E. A. Cornell, and C. Wieman, Physical Review Letters 78, 586 (1997).
  * Ancilotto  _et al._ (2018) F. Ancilotto, M. Barranco, M. Guilleumas, and M. Pi, Physical Review A 98, 053623 (2018).
  * Cappellaro  _et al._ (2018) A. Cappellaro, T. Macrì, and L. Salasnich, Physical Review A 97, 053623 (2018).
  * Cheiney  _et al._ (2018) P. Cheiney, C. R. Cabrera, J. Sanz, B. Naylor, L. Tanzi, and L. Tarruell, Phys. Rev. Lett. 120, 135301 (2018).
  * Khaykovich  _et al._ (2002) L. Khaykovich, F. Schreck, G. Ferrari, T. Bourdel, J. Cubizolles, L. D. Carr, Y. Castin, and C. Salomon, Science 296, 1290 (2002).
  * Khan and Panigrahi (2013) A. Khan and P. K. Panigrahi, Journal of Physics B: Atomic, Molecular and Optical Physics 46, 115302 (2013).
  * Debnath  _et al._ (2022) A. Debnath, J. Tarun, and A. Khan, Journal of Physics B: Atomic, Molecular and Optical Physics 55, 025301 (2022).
  * Efimov (1973) V. Efimov, Nuclear Physics A 210, 157 (1973).
  * Abramowitz  _et al._ (1988) M. Abramowitz, I. A. Stegun, and R. H. Romer, “Handbook of mathematical functions with formulas, graphs, and mathematical tables,” (1988).
  * Debnath and Khan (2020) A. Debnath and A. Khan, The European Physical Journal D 74, 1 (2020).
  * Ilg  _et al._ (2018) T. Ilg, J. Kumlin, L. Santos, D. S. Petrov, and H. P. Büchler, Physical Review A 98, 051604 (2018).
  * Ferrier-Barbut  _et al._ (2016) I. Ferrier-Barbut, H. Kadau, M. Schmitt, M. Wenzel, and T. Pfau, Phys. Rev. Lett. 116, 215301 (2016).
  * Wächtler and Santos (2016) F. Wächtler and L. Santos, Phys. Rev. A 93, 061603 (2016).


# A Neural Temporal Model for Human Motion Prediction

Anand Gopalakrishnan1, Ankur Mali1, Dan Kifer1, C. Lee Giles1, Alexander G.
Ororbia2
Pennsylvania State University, University Park, PA, 168011
Rochester Institute of Technology, Rochester, NY, 146232
{aug440, aam35, duk17, clg20} @psu.edu1<EMAIL_ADDRESS>
###### Abstract

We propose novel neural temporal models for predicting and synthesizing human
motion, achieving state-of-the-art in modeling long-term motion trajectories
while being competitive with prior work in short-term prediction and requiring
significantly less computation. Key aspects of our proposed system include: 1)
a novel, two-level processing architecture that aids in generating planned
trajectories, 2) a simple set of easily computable features that integrate
derivative information, and 3) a novel multi-objective loss function that
helps the model to slowly progress from simple next-step prediction to the
harder task of multi-step, closed-loop prediction. Our results demonstrate
that these innovations improve the modeling of long-term motion trajectories.
Finally, we propose a novel metric, called Normalized Power Spectrum
Similarity (NPSS), to evaluate the long-term predictive ability of motion
synthesis models, complementing the popular mean-squared error (MSE) measure
of Euler joint angles over time. We conduct a user study to determine if the
proposed NPSS correlates with human evaluation of long-term motion more
strongly than MSE and find that it indeed does. We release code and additional
results (visualizations) for this paper at:
https://github.com/cr7anand/neural_temporal_models

##  1 Introduction

We address the problem of building predictive models of human movement using
motion capture data. Specifically, we explore models that can successfully be
used in forecasting the 3D pose of a human subject conditioned on a small,
initial history (a set of priming frames). Current work has focused on two
separate but complementary sub-tasks: 1) short-term motion prediction, which
is generally evaluated quantitatively by measuring mean squared error (MSE)
over a short horizon, and 2) long-term motion prediction, which is evaluated
qualitatively by manual, visual inspection of samples in order to evaluate
plausible trajectories of human motion over long spans of time. Short-term
models are useful in applications of motion tracking while long-term models
are useful as motion generation tools for computer graphics [24, 16, 12].
Models successful in these sub-tasks are also valuable for human gait
analysis, studies in the kinematics of human motion, and in human-computer
interaction applications [2, 25].

Solving the above two sub-problems in motion prediction is challenging given
the high dimensionality of the input data as well as the difficulty in
capturing the nonlinear dynamics and stochasticity inherent in human motion
from observations alone. Furthermore, human motion, in strong contrast to the
motion of other objects, depends on the subject’s underlying intent and high-
level semantic concepts which are tremendously difficult to model
computationally. Traditionally, models were built in the framework of expert
systems and made use of strong simplifying assumptions, such as treating the
underlying process as if it was Markovian and smooth or using low-dimensional
embeddings [29, 19]. These approaches often led to less-than-satisfactory
performance. With the modern successes of artificial neural networks [15] in
application domains ranging from computer vision [13] to machine translation
[1] and language modeling [18], many current models of motion are becoming
increasingly based on neural architectures.

In this paper, we attack the above two sub-problems using the following
strategies. First, we augment the joint angle feature vector, typically fed
into predictive neural models, with motion derivative information. This can be
easily computed using a finite-difference approximation and naturally contains
(temporally) local information that is crucial for generating smooth and
consistent motion trajectories. Furthermore, our results demonstrate that the
popular approach of training recurrent neural networks (RNNs) in an open loop,
i.e., where ground truth input data is fed in at every time step $t$ to
predict output at $t+1$, is insufficient when using these models for closed-
loop test scenarios, i.e., where model output at time step $t$ is itself used
as input to model at time step $t+1$. In the case of closed-loop generation,
the model fails to make good predictions over long time horizons due to
drifting and an accumulation of next-step error. To remedy this, we introduce
a simple, novel multi-objective loss function that balances the goals of
effective next-step prediction with generating good long-term, closed-loop
predictions, which we find greatly alleviates model drifting. The neural
architectures we propose, which make use of a novel, differentiable backward-
planning mechanism, are computationally less expensive and far simpler than
competing alternatives.

Finally, we address the dearth of effective quantitative methods for
evaluating long-term motion synthesis by proposing a novel metric we call the
Normalized Power Spectrum Similarity (NPSS). NPSS is meant to complement the
popular MSE by addressing some of its drawbacks including: a) a frequency-
shift in a predicted sequence, e.g., walking at a faster or slower rate,
compared to ground-truth will be heavily penalized by MSE despite being
qualitatively similar, and b) a phase-shift in predicted sequence, e.g., if a
few frames of motion are missed/skipped by the model, the resulting predicted
motion sequence will be phase-shifted but MSE will heavily penalize it despite
qualitative similarity with ground-truth. Our measure accounts for these
issues as it is designed to capture the difference in the power spectrum of
the ground truth frames and the model’s predicted joint angles. The key
contributions of this work include: 1) a novel, two-stage processing
architecture, 2) augmenting the input space with easily computable features
useful for the domain of motion, 3) development of a novel loss function that
can help guide the model towards generating longer-term motion trajectories
and, 4) a novel, evaluation metric called NPSS for long-term human motion
quality evaluation, which we will validate with a human user study.

##  2 Related Work

Research in motion synthesis has a long history, with many models proposed
over the years. Only in recent times have neural architectures come to the
forefront of this domain, quickly supplanting classical statistical learning
approaches and hand-crafted methods. [4] proposed two architectures: 1) the
LSTM-3LR and 2) the ERD (Encoder-Recurrent Decoder). The LSTM-3LR consists of
$3$ layers of $1000$ Long Short-Term Memory (LSTM) units whereas the ERD model
uses $2$ layers of $1000$ LSTM units and nonlinear multilayer feedforward
networks for encoding and decoding. However, the authors observed that, during
inference, the models would quickly diverge and produce unrealistic motion.
They alleviate this by gradually adding noise to the input during training
which helps in generating plausible motion over longer horizons. [10] proposed
the Structural-RNN (SRNN), which takes a manually designed spatio-temporal
graph and converts it into a multilayer RNN architecture with body RNNs being
assigned to model specific body parts and edge-RNNs to model interactions
between body parts. This work also uses the noise scheduling technique
employed earlier by [4] to alleviate drifting. They show that their network
outperforms previous methods in both short-term motion prediction as well as
long-term qualitative motion. Recently, [17] proposed simple but hard-to-beat
baselines on short-term motion prediction as well as a 1-layer sequence-to-
sequence (seq2seq) model [26] with 1024 Gated Recurrent Unit (GRU) units and a
linear output decoder for short-term and long-term motion prediction.
Additionally, they trained their long-term model using a sampling-loss as a
simpler alternative to noise scheduling in order to alleviate drifting. More
recently, [5] proposed a model that couples a denoising autoencoder and an
LSTM-3LR network to alleviate drifting for long-term motion synthesis.
However, a drawback of their approach is that both the autoencoder and
LSTM-3LR networks are first pre-trained independently followed by a subsequent
joint fine-tuning step.

##  3 A Neural Motion Synthesizer

In this section, we will describe our neural system for motion synthesis,
which integrates a novel architecture with a novel loss function and useful,
easily computable features. Since our focus is on the specific problem of
motion synthesis, we will start by first detailing the benchmark we will test
our models against.

###  3.1 Data and Preprocessing

Staying consistent with previous work on human motion synthesis [4, 10], we
use the Human 3.6 Million (h3.6m) dataset [9], which is currently the largest
publicly available motion capture (mocap) database. The h3.6m dataset consists
of $7$ actors performing $15$ different actions. Previous work [4, 10, 17] has
been particularly focused on $4$ out of these $15$ categories, e.g., walking,
eating, smoking, and discussion when evaluating model performance. To create
the test-set, we follow prior work by extracting $8$ motion sequences per
action type from subject #5, yielding the exact same 32 test sequences as used
in [4, 10]. The remaining sequences for subject #5 are then placed into a
validation subset that is used for tuning hyper-parameters. The data of the
other six subjects is then used as a training set. We furthermore adopt the
pose representation and evaluation metrics as used previously in [4, 10] to
facilitate experimental comparability. Pose is represented as an exponential
map of each joint (refer to [27] for further details). To evaluate our models,
we measure the Euclidean distance between predictions and ground truth in
Euler angle-space at various time slices along the predicted sequence.

###  3.2 Architecture

The architecture we propose for human motion prediction and synthesis we call
the Verso-Time Label Noise-RNN model (VTLN-RNN), which consists of a top-level
and a bottom-level RNN. Combined, the two RNNs have fewer parameters than
prior motion deep learning motion synthesis models. The top-level RNN is meant
to serve as a learnable noise process inspired by the work of [8], which runs
backwards in time, starting from a sampled initial hidden state ($z_{\phi}$)
and is conditioned on the one-hot encoding of the action label. This noise
process is used to generate a sequence of $K$ “guide vectors” (where $K$ is
the number of future frames we want to predict, or the prediction horizon)
that will be subsequently used by the lower-level RNN. The lower-level RNN, or
the Body-RNN, runs forward in time, taking in as input at each time step the
joint angle vector $\mathbf{x}_{t}$ as well as the corresponding guide vector
$\mathbf{p}_{t}$ to generate a prediction of the mocap angles for time-step
$t+1$. In essence, running the VTLN-RNN entails using the top-level noise
process RNN to generate the guide vectors and then using the Body-RNN to
integrate both the bottom-up mocap input vectors and the top-down guide
vectors to compute the final hidden states $\mathbf{h}_{t}$ and the next-step
predictions $\mathbf{\hat{x}}_{t}$. The unrolled model is depicted in Figure
1. The loss is computed using the Body-RNN’s predicted outputs and the
corresponding ground-truth mocap vectors.

In order to sample the initial hidden state of the top-level noise process, we
first structure it to work like a multivariate Gaussian distribution, drawing
inspiration from the re-parameterization trick [11] and the adaptive noise
scheme proposed in [8]. The initial state $z_{\phi}$ of the top-level noise
process is computed as $z_{\phi}=\mu+\Sigma\otimes\epsilon$, where
$\epsilon\sim\mathcal{N}(0,\textrm{I})$, $\mu$ the mean of the random
variable, and $\Sigma$ is its covariance, specifically a diagonal covariance.
$\mu$ and $\Sigma$ are parameters that are learned along with the rest of the
neural network weights using back-propagated gradients during training. This
formulation of the hidden state allows the designer to input samples from a
simple base distribution, e.g., a standard Gaussian, instead of having to tune
the noise parameters, such as its variance, by hand.

Figure 1: VTLN-RNN architecture

In this paper, we use the Gated Recurrent Unit (GRU) [3] to instantiate both
the top-level and bottom-level RNNs of the VTLN-RNN due to its simplicity,
competitive performance, and ease of training compared to the LSTM [7]. The
cell update equations for the top-level GRU remain the same as described in
[3] except that we note its non-state input is the action label (which remains
fixed over the length of the sequence). However, the cell update equations for
the Body-GRU are as follows:

$\displaystyle r_{j}=$
$\displaystyle\sigma\big{(}[W_{r}x_{t}]_{j}+[U_{r}h_{t-1}]_{j}+[V_{r}p_{t}]_{j}\big{)}$
(1) $\displaystyle z_{j}=$
$\displaystyle\sigma\big{(}[W_{z}x_{t}]_{j}+[U_{z}h_{t-1}]_{j}+[V_{z}p_{t}]_{j}\big{)}$
(2) $\displaystyle\tilde{h_{j}}^{t}=$
$\displaystyle\Phi\big{(}[Wx_{t}]_{j}+[U(r\otimes
h_{t-1})]_{j}+[Vp_{t}]_{j}\big{)}$ (3) $\displaystyle h_{j}^{t}=$
$\displaystyle z_{j}\otimes
h_{j}^{t-1}+(1-z_{j})\otimes\tilde{h_{j}}^{t}\mbox{.}$ (4)

The motivation behind the VTLN-RNN structure was to hierarchically decompose
the motion synthesis problem into a two-level process, much as has been
successfully done in neural-based dialogue modeling [23, 22]. The top-level
RNN would serve to roughly sketch out a course trajectory that the lower-level
RNN would take in, further conditioned on actual data and its own internal
state. However, unlike the hierarchical neural dialogue models that served as
inspiration, our top-level process runs in the opposite (temporal) direction
of the data itself, i.e., backwards. We chose to do this considering gradient
flow in the unrolled computation graph. If the top-level process starts at
time $t=K$ and works backward to time $t=1$, the parameter updates of the top-
level model will depend more heavily on information from the future (or from
far later on in a sequence) and this information would be encoded in the
synaptic weights related to a specific action type/label. When the top-level
process is used to generate the sequence of guide vectors, it creates “hints”
or coarsely defined states that the lower-level Body-RNN can then refine based
on actual input data or its own closed-loop predictions.

While it is hard to prove that the top-level RNN is truly “planning” out the
ultimate trajectory of the model’s predictions, our experiments will show that
our two-level process offers some useful regularization, improving model
generalization over mechanisms such as drop-out. Additionally, our model used
for both short-term and long-term motion prediction has significantly fewer
parameters compared to [4, 10, 17] and yet achieves state-of-art results as
shown in Table 7 on long-term motion prediction. Furthermore, it is
competitive with [17] state-of-art results on short-term motion prediction as
shown in Table 6.

Models | No. of parameters
---|---
ERD [4] | 14,842,054
LSTM-3LR [4] | 20,282,054
SRNN [10] | 18,368,534
MBR-long [17] | 3,425,334
GRU-d (ours) | 2,735,670
VGRU-d (ours) | 3,413,047
Table 1: Number of parameters of models.

###  3.3 Incorporating Derivative Information

Motion derivatives contain crucial feature information used to model local
(near past) motion information. These features are cheap to compute and do not
require any additional model parameters. Motivated by this, we extract motion
derivatives by a using a finite backward difference approximation, calculated
as follows:

$\nabla_{h}^{n}[f](x)=\sum_{i=0}^{n}(-1)^{n}{n\choose i}f(x-ih)$ (5)

where $i$ indexes the order of the derivative we would like to approximate, up
to $n$, and $h$ is a non-zero spacing constant.
We extract the $n=\\{1,2,3\\}$ motion derivatives with $h=1$ using the above
equation and append these vectors to the vector of joint angles. The linear
decoder of our recurrent model outputs only joint angles for the next time
step. During closed loop, iterative multi-step prediction we calculate these
motion derivatives on-the-fly.

###  3.4 Facilitating Closed-Loop Prediction

The standard way to train RNNs for sequence prediction tasks is to feed the
ground truth inputs at every time step during training. Then, at test time,
the model’s previous prediction at $t$ is fed in, treating it as it were
ground truth input, when making a prediction at $t+1$. This process is known
as closed-loop (or iterative) prediction. However, a key issue with this
method is that the model is unable to recover from the accumulation of errors
leading to a significant degradation in the RNN’s predictions over time. This
is due to the strong mismatch in the inputs presented during training (i.e.
ground-truth inputs) and test time (i.e. the model’s own noisy predictions
from previous time steps). This causes synthesized long-term motion
trajectories to quickly diverge from the manifold of plausible motion
trajectories. As mentioned earlier, [4] and [10] alleviate this issue by
injecting Gaussian noise, of gradually increasing magnitude, to inputs during
training. [17] used a sampling loss where, during training, the model outputs
are fed back to itself. Professor Forcing [14] addresses this issue by using
an adversarial training regime to ensure the hidden states of the RNN are
constrained to be similar during train and test time. However, this method is
computationally expensive, needs careful hyperparameter tuning, and suffers
from stability issues normally encountered in the training of Generative
Adversarial Networks. More recently, [31] showed that their method, or auto-
conditioning, helps the RNN models produce good qualitative long-term motion
by alternating between feeding in ground-truth samples and the model’s own
outputs during training.

We view the problem of using the RNN for multi-step iterative prediction at
test time from the perspective of multi-task and curriculum learning. We
ultimately require the RNNs to achieve good performance on the hard-task of
multi-step iterative prediction starting from the simple task of one-step
prediction. An intuitive way to achieve this would be to gradually make the
RNN progress from the simple task of one-step prediction (ground truth fed in
at every time step) to the final goal of multi-step iterative prediction.
Defining a composite loss function with separate terms for measuring one-step
prediction and multi-step iterative prediction losses, and weighting these
terms, would ensure that the network slowly adapts from being able to only
predict one-step ahead to becoming capable of multi-step iterative prediction
during the course of the training cycle. This intuition forms the basis of our
multi-objective loss function defined as follows,

$L(\widehat{y},y)=\frac{1}{T}\sum_{t=0}^{T}(\widehat{y_{o}}^{t}-y^{t})^{2}+\frac{\lambda}{T^{\prime}}\sum_{t_{1}=0}^{T^{\prime}}(\widehat{y_{c}}^{t_{1}}-y^{t_{1}})^{2}$
(6)

where $y^{t}$ = ground-truth output at $t$, $\widehat{y_{o}}^{t}$ = model
output in open-loop mode at $t$, $\widehat{y_{c}}^{t_{1}}$ = model output in
closed-loop mode at $t_{1}$. Open-loop mode refers to feeding ground-truth
inputs at every time step to the RNN in order to produce outputs and closed-
loop mode refers to feeding the model’s own output at $t$ as input to it at
$t+1$. For every input sequence, this loss requires us to run the forward pass
twice: 1) compute $\widehat{y_{o}}^{t}$ in open-loop mode, and 2) compute
$\widehat{y_{c}}^{t}$ in closed-loop mode. We gradually increase $\lambda$
using a step schedule over the training cycle starting from zero at the
beginning. This schedule therefore gradually places greater importance on the
loss-term focused on closed-loop predictions as the network learns to make
better one-step predictions. In our long-term motion synthesis experiments, we
will see that our loss outperforms noise scheduling [4, 10], auto-conditioning
[31], and the sampling loss of [17].

###  3.5 A Complementary Long-Term Motion Metric

The use of mean-squared error (MSE) as an evaluation metric for models has
been standard practice [4, 10, 17] for both short-term motion prediction and
long-term motion synthesis tasks. In short-term motion prediction, the metric
needs to capture how well a model mimics ground-truth data over short-term
horizons (i.e 0-500 milliseconds) since it is to be used in motion tracking
applications.

However in the long-term motion synthesis task, models need to be evaluated on
how well they generate plausible future motion over long-term horizons given
some seed frames of motion. Since human motion is inherently stochastic over
long time horizons, models can significantly deviate from the ground-truth
trajectories and have a large MSE despite producing qualitatively good human
motion. This problem has been noted in prior work [4, 10, 17]. There are a
variety of causes. For example, if the predictions correspond to walking at a
slower pace, the joint angles will be misaligned (frequency-shift) and MSE
computed will diverge over time. In the short term, the joint angles may still
be similar enough for MSE to meaningfully capture similarity, but in the long
term they will become significantly different. Similarly, adding or removing a
few extra frames of motion (phase-shift) compared to ground-truth sequence
will result in high MSE values because frames are, again, misaligned.
Therefore, as noted in prior work [4, 10, 17], the use of MSE as an evaluation
metric is not appropriate for the long-term task. However, no attempt has
previously been made to suggest another quantitative metric for evaluating
long-term motion synthesis models.

In this paper, we propose such a metric, backed by a user study, based on the
following intuition. We can say that the qualitative essence of any action,
e.g., walking, eating, running, can be captured through the frequency
signature of joint angles of the body while performing that action. Take
walking at a slower pace as an example – the power spectrum (obtained from a
discrete Fourier transform) would show spikes at a slightly lower frequency
and the addition or removal of a few frames would show up as a phase-shift in
the frequency domain. The examples of slow/fast or phase-shifted walking
involve periodic sub-actions, whereas aperiodic actions, such as discussion,
will show a more uniform spread of power in the frequency domain (this
indicates a lack of periodicity in the action which is also being picked up by
the power spectrum). Measuring similarity of the power spectrum between
between a ground truth sequence and a corresponding generated sequence for the
same motion type would account for these phenomena and correlate better with
the visual quality (see user study results in Section 4.2) of samples. The
field of content-based image retrieval have used Earth Mover’s Distance (EMD)
[21, 6] to quantify perceptual similarity of images using the EMD between
their color histograms. Based on the intuition we have developed so far and
the recent successful application of EMD, we propose an EMD-based metric over
the power spectrum that overcomes many of the shortcomings of MSE as an
evaluation metric on the long-term task.

For a given action class in the test set, let there be $k$ sequences each of
$T$ length and output vector of joint angles at each time-step be $D$
dimensional. We define $x_{i,j}[t]$ to be the ground-truth value at time $t$
for $j^{th}$ feature dimension for $i^{th}$ sequence and $y_{i,j}[t]$ to be
the corresponding model prediction. Also, let $X_{i,j}[f]$ and $Y_{i,j}[f]$ be
the squared magnitude spectrum of Discrete Fourier Transform coefficients (per
sequence $i$ per feature dimension $j$ ) of $x_{i,j}[t]$ and $y_{i.j}[t]$
respectively. First we normalize $X_{i,j}[f]$ and $Y_{i,j}[f]$ w.r.t $f$ as,

$\displaystyle X_{i,j}^{\textrm{norm}}[f]=$
$\displaystyle\frac{X_{i,j}[f]}{\sum_{f}X_{i,j}[f]};Y_{i,j}^{\textrm{norm}}[f]=\frac{Y_{i,j}[f]}{\sum_{f}Y_{i,j}[f]}$
(7) $\displaystyle\textrm{emd}_{i,j}=$
$\displaystyle\|X_{i,j}^{\textrm{norm}}[f]-Y_{i,j}^{\textrm{norm}}[f])\|_{1}$
(8)

where, $\|.\|_{1}$ is the L1-norm. Finally, we use a power weighted average
over all $i$ and $j$ of 1-D EMD distances computed in (8) as shown below,

$NPSS=\frac{\sum_{i}\sum_{j}p_{i,j}*\textrm{emd}_{i,j}}{\sum_{i}\sum_{j}p_{i,j}}\quad
p_{i,j}=\sum_{f}X_{i,j}^{\textrm{norm}}[f]$ (9)

where $p_{i,j}$ = total power of $i^{th}$ feature in the $j^{th}$ sequence, to
arrive at our scalar evaluation metric for an evaluation set of sequences for
a given action class. We refer to our metric as normalized power spectrum
similarity (NPSS). Another interpretation is that we can view long-term motion
synthesis as a generative modeling task. By this interpretation, the
evaluation metric must capture differences in the distributions of the ground-
truth and predicted motion samples. NPSS captures distributional differences
in the power spectrum of joint angles of the ground-truth and predicted
sequences. As a result, it is better equipped to model differences in the
visual quality of motion trajectories.

##  4 Experiments

###  4.1 Training Setup

For our short-term model, the VGRU-r1 (MA), we trained on all action classes
using our proposed multi-objective cost, calculating gradients over mini-
batches of $32$ samples (clipping gradient norms to $5$) and optimizing
parameters over $100,000$ iterations RMSprop [28] with initial learning rate
$\lambda=0.0001$ and decayed by $0.8$ every 5000 iterations until 60,000
iterations. Drop-out [30, 20], with probability of 0.3, was applied only to
the Body-RNN, which was further modified to use skip connections that connect
input units to output units, as in [17]. The model was given $50$ seed frames
and tasked with predicting the next $10$ subsequent frames ($400$
milliseconds). When training for this, the VTLN-RNN is unrolled backwards
while the Body-RNN is unrolled forwards, in time, over $60$ steps. (Note: MA
stands for multi-action, SA for single-action.)

For our long-term models, which were trained on single-action data, parameter
optimization was carried out with RMSprop ($\lambda=0.0002$, decayed by $0.6$
every $2000$ iterations) over $10,000$ iterations with mini-batches of 32,
using, again, our proposed cost function. Models were fed in $50$ seed frames
and made to predict the next $100$ frames ($4$ sec), which meant that the
VTLN-RNN was unrolled backwards and the Body-RNN forwards $150$ steps. The
input vector to the Body-RNN consisted of joint angles appended with motion
derivatives. VGRU-d refers to our proposed VTLN-RNN architecture where the
VTLN-RNN and Body-RNN both contain only a single layer of 512 GRU cells. GRU-d
refers to a 2-layer GRU model (512 units in each). Both VGRU-d and GRU-d
models are trained with our proposed loss and make use of inputs augmented
with motion derivatives. VGRU-ac refers to our VTLN-RNN architecture trained
with auto-conditioning [31], using the recommended length of $5$, serving as a
baseline. For all models (short and long-term), hyper-parameters were tuned on
a separate validation set.

Figure 2: Long-term motion synthesis on walking activity on test sequence.
Snapshots are shown at 160, 560, 1000, 2000 and 4000 milliseconds (from top-
to-bottom) along the prediction time-axis. We see that GRU-d and VTLN-GRU-d
are qualitatively closer to the ground-truth sequence than MBR-long and VTLN-
GRU-ac.

###  4.2 User Study: Long-Term Motion Synthesis

We conducted a user study to understand how human judgment of long-term motion
correlates with MSE as well as our proposed NPSS. A desirable quantitative
evaluation metric for long-term human motion would be one that strongly agrees
with human judgment. In order to conduct this study, we considered the 6
models from Table 4 (i.e. VGRU-r1(SA), MBR-unsup (SA), MBR-long, VGRU-ac,
GRU-d and VGRU-d). In each trial, a user was presented videos of the ground-
truth motion and corresponding model predictions from a randomly chosen pair
of models (from the list above) for a given test-set action sequence (the
ordering of the models was random and identities were hidden from the users).
Users were asked to compare model-generated motion trajectories with the
ground truth, based on which one possessed better “motion quality”. The users
were informed that the phrase “motion quality” referred to
similarity/closeness in overall skeletal pose (i.e. overall posture) and joint
motion dynamics over the entire sequence, rather than simple point-to-point
matches in time, and made their decisions based on this criteria. Please refer
to the supplementary material for a sample screenshot of the user survey
video.

For each of the $4$ action classes (i.e. walking, eating, smoking, and
discussion) we presented $20$ video sequences of the ground-truth with the A
versus B comparison (refer to supplementary for a sample user study
screenshot). Video samples were selected uniformly and randomly (without
replacement) from all possible, pairwise combinations of the $6$ models. We
then selected a test sequence for an action class, via uniform random sampling
with replacement, and presented the ground-truth motion sequence and
previously picked paired model predictions for that sequence. This process is
repeated to generate 20 videos (i.e. 20 questions) for each of the $4$
actions. The study involved 20 participants for each of the 4 action class
surveys.

Now for the 2 evaluation metrics (i.e. MSE and NPSS) we derive rankings of the
models used in the user study. For the MSE metric ranking, we compute the sum
of MSE over all time slices for the long-term window (i.e. 80, 160, 320, 400,
560, 1000 milliseconds, which is consistent with prior work [10]). For NPSS,
we use the results in Table 4 to arrive at a ranking of the models for all $4$
actions. Then, we use these rankings (MSE and NPSS) for each action class to
make predictions for each question in the user survey. As shown in Table 3, we
compute the probabilities of agreement and disagreement with users for MSE and
NPSS.

Metrics | Walking | Eating | Smoking | Discussion
---|---|---|---|---
MSE rankings | 1\. VGRU-r1(SA) | 1\. VGRU-r1(SA) | 1\. VGRU-r1(SA) | 1\. VGRU-r1(SA)
2\. MBR-unsup (SA) | 2\. VGRU-d | 2\. MBR-unsup (SA) | 2\. VGRU-d
3\. VGRU-d | 3\. MBR-unsup (SA) | 3\. VGRU-d | 3\. GRU-d
4\. MBR-long | 4\. VGRU-ac | 4\. VGRU-ac | 4\. VGRU-ac
5\. VGRU-ac | 5\. GRU-d | 5\. GRU-d | 5\. MBR-unsup (SA)
6\. GRU-d | 6\. MBR-long | 6\. MBR-long | 6\. MBR-long
NPSS rankings | 1\. VGRU-d | 1\. GRU-d | 1\. VGRU-d | 1\. VGRU-ac
2\. GRU-d | 2\. VGRU-ac | 2\. GRU-d (SA) | 2\. GRU-d
3\. VGRU-ac | 3\. VGRU-d (SA) | 3\. VGRU-ac | 3\. VGRU-d
4\. VGRU-r1 (SA) | 4\. VGRU-r1 (SA) | 4\. VGRU-r1 (SA) | 4\. MBR-unsup (SA)
5\. MBR-long | 5\. MBR-unsup (SA) | 5\. MBR-unsup (SA) | 5\. MBR-long
6\. MBR-unsup (SA) | 6\. MBR-long | 6\. MBR-long | 6\. VGRU-r1 (SA)

Table 2: Long-term motion model MSE & NPSS rankings. | MSE | NPSS
---|---|---
Agree | 0.4875 (39/80) | 0.8125 (65/80)
Disagree | 0.5125 (41/80) | 0.1875 (15/80)
Table 3: User agreement ratios for MSE & NPSS aggregated across all actions
taking majority user vote as ground-truth. a/b = number of times user answers
agrees with metric’s answer/ total equences in user survey across 4 actions.

Furthermore, we conducted a Binomial test of proportions to test the claim
that NPSS agrees better with user judgment than MSE. In this test, $p_{1}$ is
defined to be the probability that, on a random sample, NPSS will agree with
human ordering/choice while $p_{2}$ is the probability that MSE will agree
with human ordering/choice. We take the null hypothesis to be $H_{0}:p_{1}\leq
p_{2}$ and the alternative hypothesis to be $H_{A}:p_{1}>p_{2}$ and seek to
test the null against the alternative hypothesis. Scientific studies typically
set the threshold of statistical significance for p-values to be below $0.01$
(smaller p-values would better support the claim that NPSS is a better metric,
confirming that $p_{1}$ is statistically larger than $p_{2}$). The value we
obtained is significantly lower than this threshold, i.e., a p-value of
$\bf{1.7}\times\bf{10^{-5}}$.

##  5 Results and Discussion

Given the results of our user study, we argue that NPSS should be preferred
(over MSE) for measuring model generation quality over long sequences (for
predictions made over longer horizons). However, to holistically evaluate a
motion synthesis model, we recommend using NPSS in tandem with MSE when
evaluating a model’s ability to make both short-term and long-term
predictions. The results of our user study for NPSS is promising, however,
further studies should be conducted to further validate and strengthen our
findings.

For compatibility with prior work, Table 7 compares the MSE of Euler angles,
measured at particular time slices on test sequences, reporting results for
competing methods such as LSTM-3LR and ERD [4], SRNN by [10], and MBR-long
[17]. Although our short-term model, VGRU-r1 (SA), displays the best
performance (lowest mse) until the $1$ second mark, it has been noted by [10,
17], and further corroborated by the results of our user study, that MSE is
not appropriate for the task of long-term motion synthesis. Table 4 shows the
NPSS metric results for models evaluated on the test set.

Models | Walking | Eating | Smoking | Discussion
---|---|---|---|---
VGRU-r1(SA) (ours) | 1.217 | 1.312 | 1.736 | 4.884
MBR-unsup (SA) [17] | 1.809 | 1.481 | 2.794 | 2.258
MBR-long [17] | 1.499 | 1.621 | 4.741 | 2.882
VGRU-ac | 1.032 | 0.842 | 1.426 | 1.651
GRU-d (ours) | 0.931 | 0.836 | 1.274 | 1.688
VGRU-d (ours) | 0.887 | 0.846 | 1.235 | 1.777

Table 4: Test-set NPSS scores (lower is better). Above the double line: short-
term models, i.e., MBR-unsup (SA), MBR-unsup. (MA) [17] (re-trained on single-
action), and ours, sampled for long-term durations. Below the line: long-term
models, i.e., MBR-long (SA) [17], and ours, such as GRU-d, VGRU-d, & VGRU-ac.

Models | Short-Term
---|---
| Walking | Eating | Smoking | Discussion
VGRU-r1 (SA) (ours) | 0.120 | 0.091 | 0.052 | 0.258
MBR-unsup (SA) [17] | 0.238 | 0.249 | 0.183 | 0.416
MBR-long [17] | 0.161 | 0.214 | 0.265 | 0.703
VGRU-ac | 0.118 | 0.113 | 0.075 | 0.256
GRU-d (ours) | 0.127 | 0.095 | 0.126 | 0.185
VGRU-d (ours) | 0.117 | 0.121 | 0.084 | 0.194
| Medium-Term
VGRU-r1 (ours)(SA) | 0.194 | 0.093 | 0.079 | 0.375
MBR-unsup (SA) [17] | 0.206 | 0.178 | 0.237 | 0.439
MBR-long [17] | 0.237 | 0.160 | 0.405 | 0.477
VGRU-ac | 0.188 | 0.103 | 0.097 | 0.298
GRU-d (ours) | 0.170 | 0.096 | 0.083 | 0.258
VGRU-d (ours) | 0.179 | 0.080 | 0.067 | 0.331
| Long-Term
VGRU-r1 (SA) (ours) | 0.544 | 0.764 | 0.948 | 2.72
MBR-unsup (SA) [17] | 0.884 | 0.684 | 1.077 | 0.943
MBR-long [17] | 0.549 | 0.754 | 1.403 | 1.245
VGRU-ac | 0.460 | 0.459 | 1.051 | 0.811
GRU-d (ours) | 0.406 | 0.332 | 0.723 | 0.785
VGRU-d (ours) | 0.359 | 0.288 | 0.577 | 1.001

Table 5: NPSS at 3 different time scales i.e 1) short-term: 0-1 second 2)
medium-term: 1-2 seconds 3) long-term: 2-4 seconds window prediction on test
set

We can see that the short-term models, VGRU-r1 (SA) and MBR-unsup (SA),
despite having the lowest MSE values (until 1 second, as in Table 7), achieve
worse scores in terms of NPSS when compared to long-term models. This result
is in accordance with the visual quality of samples produced by these models
and illustrates how NPSS is better equipped to capture differences in sample
quality than MSE. Based on the NPSS metric, VGRU-d and GRU-d produce better
long-term motion trajectories, outperforming MBR-long and VGRU-ac across all
$4$ action classes.

| Walking | Eating | Smoking | Discussion
---|---|---|---|---
milliseconds | 80 | 160 | 320 | 400 | 80 | 160 | 320 | 400 | 80 | 160 | 320 | 400 | 80 | 160 | 320 | 400
Zero-velocity [17] | 0.39 | 0.68 | 0.99 | 1.15 | 0.27 | 0.48 | 0.73 | 0.86 | 0.26 | 0.48 | 0.97 | 0.95 | 0.31 | 0.67 | 0.94 | 1.04
MBR-unsup (MA) [17] | 0.27 | 0.47 | 0.70 | 0.78 | 0.25 | 0.43 | 0.71 | 0.87 | 0.33 | 0.61 | 1.04 | 1.19 | 0.31 | 0.69 | 1.03 | 1.12
MBR-sup (MA) | 0.28 | 0.49 | 0.72 | 0.81 | 0.23 | 0.39 | 0.62 | 0.76 | 0.33 | 0.61 | 1.05 | 1.15 | 0.31 | 0.68 | 1.01 | 1.09
VGRU-r1 (MA) (ours) | 0.34 | 0.47 | 0.64 | 0.72 | 0.27 | 0.40 | 0.64 | 0.79 | 0.36 | 0.61 | 0.85 | 0.92 | 0.46 | 0.82 | 0.95 | 1.21
|  $\pm$ 1e-3 |  $\pm$ 1e-3 |  $\pm$ 2e-3 |  $\pm$ 2e-3 |  $\pm$ 2e-3 |  $\pm$ 1e-3 |  $\pm$ 2e-3 |  $\pm$ 2e-3 |  $\pm$ 6e-4 |  $\pm$ 1e-3 |  $\pm$ 1e-3 |  $\pm$ 1e-3 |  $\pm$ 2e-3 |  $\pm$ 1e-3 |  $\pm$ 3e-3 |  $\pm$ 5e-3

Table 6: Short-term results: MSE on test sequences for short-term motion
prediction. All models are trained on multiple actions. VGRU-r1 (MA) refers to
our VTLN-RNN with 1 layer (512 GRU unit) and a Body-RNN with 1 layer 512 GRU
cells, where Body-RNN has residual input-to-output connections as in [17]. For
the VGRU-r1, model we compute mean and standard error over $30$ trials.

| Walking | Eating | Smoking | Discussion
---|---|---|---|---
models | 80 | 160 | 320 | 400 | 560 | 1000 | 80 | 160 | 320 | 400 | 560 | 1000 | 80 | 160 | 320 | 400 | 560 | 1000 | 80 | 160 | 320 | 400 | 560 | 1000
MBR-unsup (SA) [17] | 0.37 | 0.655 | 0.987 | 1.095 | 1.286 | 1.476 | 0.411 | 0.781 | 1.375 | 1.630 | 1.926 | 2.106 | 0.472 | 0.891 | 1.497 | 1.726 | 2.077 | 2.581 | 0.701 | 1.326 | 2.134 | 2.433 | 2.996 | 2.950
VGRU-r1 (SA) (ours) | 0.410 | 0.570 | 0.807 | 0.868 | 1.026 | 1.231 | 0.285 | 0.441 | 0.668 | 0.829 | 0.995 | 1.531 | 0.378 | 0.656 | 0.916 | 0.994 | 1.147 | 1.837 | 0.504 | 0.909 | 1.074 | 1.282 | 1.653 | 2.168
|  $\pm$ 1e-3 |  $\pm$ 1e-3 |  $\pm$ 2e-3 |  $\pm$ 3e-3 |  $\pm$ 3e-3 |  $\pm$ 3e-3 |  $\pm$ 2e-3 |  $\pm$ 2e-3 |  $\pm$ 2e-3 |  $\pm$ 3e-3 |  $\pm$ 3e-3 |  $\pm$ 3e-3 |  $\pm$ 1e-3 |  $\pm$ 1e-3 |  $\pm$ 1e-3 |  $\pm$ 2e-3 |  $\pm$ 2e-3 |  $\pm$ 2e-3 |  $\pm$ 1e-3 |  $\pm$ 2e-3 |  $\pm$ 4e-3 |  $\pm$ 5e-3 |  $\pm$ 6e-3 |  $\pm$ 7e-3
ERD [4] | 1.30 | 1.56 | 1.84 | - | 2.00 | 2.38 | 1.66 | 1.93 | 2.28 | - | 2.36 | 2.41 | 2.34 | 2.74 | 3.73 | - | 3.68 | 3.82 | 2.67 | 2.97 | 3.23 | - | 3.47 | 2.92
LSTM-3LR [4] | 1.18 | 1.50 | 1.67 | - | 1.81 | 2.20 | 1.36 | 1.79 | 2.29 | - | 2.49 | 2.82 | 2.05 | 2.34 | 3.10 | - | 3.24 | 3.42 | 2.25 | 2.33 | 2.45 | - | 2.48 | 2.93
SRNN [10] | 1.08 | 1.34 | 1.60 | - | 1.90 | 2.13 | 1.35 | 1.71 | 2.12 | - | 2.28 | 2.58 | 1.90 | 2.30 | 2.90 | - | 3.21 | 3.23 | 1.67 | 2.03 | 2.20 | - | 2.39 | 2.43
VGRU-ac | 1.180 | 1.210 | 1.247 | 1.236 | 1.291 | 1.363 | 1.150 | 1.210 | 1.310 | 1.400 | 1.490 | 1.700 | 1.81 | 1.950 | 2.080 | 2.140 | 2.240 | 2.440 | 1.720 | 1.970 | 1.930 | 1.870 | 2.050 | 2.147
|  $\pm$ 3e-4 |  $\pm$ 3e-4 |  $\pm$ 2e-4 |  $\pm$ 3e-4 |  $\pm$ 6e-4 |  $\pm$ 7e-4 |  $\pm$ 2e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4
MBR-long [17] | 0.93 | 1.05 | 1.24 | 1.29 | 1.43 | 1.56 | 1.13 | 1.35 | 1.75 | 1.91 | 2.07 | 2.28 | 1.29 | 2.07 | 2.53 | 2.56 | 2.76 | 3.39 | 1.63 | 2.03 | 2.57 | 2.72 | 2.96 | 2.94
GRU-d (ours) | 1.311 | 1.333 | 1.369 | 1.364 | 1.350 | 1.370 | 1.275 | 1.305 | 1.386 | 1.466 | 1.530 | 1.702 | 1.943 | 2.062 | 2.201 | 2.255 | 2.342 | 2.486 | 1.744 | 1.980 | 2.026 | 1.994 | 2.214 | 2.172
VGRU-d (ours) | 1.108 | 1.146 | 1.211 | 1.200 | 1.220 | 1.280 | 1.090 | 1.160 | 1.240 | 1.330 | 1.370 | 1.560 | 1.670 | 1.800 | 1.940 | 1.980 | 2.060 | 2.320 | 1.749 | 2.037 | 2.011 | 1.868 | 2.088 | 2.318
|  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 2e-4 |  $\pm$ 2e-4 |  $\pm$ 3e-4 |  $\pm$ 2e-4 |  $\pm$ 2e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4 |  $\pm$ 1e-4

Table 7: Long-term motion results: All models are trained on single-action
data (SA = single-action). Top set show short-term models including the (MBR-
unsup(SA) = Residual unsup. (MA) from [17] re-trained on SA) and ours sampled
for longer duration to match long-term duration. Bottom set shows long-term
models by MBR-long = sampling-based loss (SA) from [17], ERD and LSTM-3LR from
[4], SRNN from [10]), our GRU-d and VGRU-d and VGRU-ac. Since the VTLN-RNN
architecture samples from a noise distribution for each forward pass, table
shows mean and standard deviation over 30 trials.

In order to discern the strengths and weaknesses of short-term and long-term
models, we computed the NPSS metric on test sequences at 3 different
timescales, i.e., 1) short-term: 0-1 s, 2) medium-term: 1-2 s, 3) long-term:
2-4 s, along the prediction timeline for test sequences shown in Table 5.
Observe that the short-term models (above the double line) VGRU-r1 (SA) and
MBR-unsup (SA) perform competitively with long-term models (below the double
line) in the short-term timescale. In the medium-term prediction horizon, the
short-term models degrade slightly more than the long-term models, as
evidenced by a small gap in the measured NPSS values. However, in the long-
term prediction horizon (of 2-4 s), the short-term models degrade
significantly relative to the long-term models. This is evidenced by wider
gaps in NPSS values. GRU-d and VGRU-d models perform best across all actions
and time-horizons, effectively outperforming MBR-long and VGRU-ac.
Finally, Table 6 shows MSE results for short-term motion prediction
experiments on multi-action data on test set sequences. Zero-velocity is a
simple, yet hard-to-beat baseline, introduced in [17], which uses the previous
frame as the prediction for current one. We can see that VGRU-r1 model is
competitive with the state-of-art short-term MBR model as well the quite
powerful, zero-velocity baseline. These results show that our proposed VTLN-
RNN architecture, augmented with motion-derivative features and our novel
multi-objective loss function, can serve as useful predictors of short-term
motion prediction as well as powerful long-term motion synthesizers. Since our
models, particularly the long-term ones, contain multiple innovations, we
conducted an ablation study to test the utility of each of proposed
components: 1) the 2-level processing VTLN-RNN architecture itself (i.e., the
backward and forward processing layers), 2) appending a vector of approximate
derivatives of joint angles as features, and 3) the multi-objective cost
function for parameter optimization. Our ablative study examined multiple
variants including: 1) a full VTLN-RNN architecture with derivative features
and the multi-objective cost, 2) a full VTLN-RNN architecture with derivative
features but without the new cost, 3) a regular RNN model (without the 2-level
processing component) with derivative features, and 4) a regular RNN (without
the 2-level processing and without appending derivatives as features or use of
the proposed cost). Details of this study are provided in the supplementary.
In short, our study revealed that each of these innovations were necessary in
improving the performance of the long-term synthesis model, with the cost
playing the most important role.

##  6 Conclusions and Future Work

For human motion prediction and synthesis, we introduced the VTLN-RNN
architecture, which uses motion derivative features as well as a novel multi-
objective loss function to achieve state-of-art performance on long-term
motion synthesis. The proposed framework also achieves competitive performance
on short-term motion prediction thereby demonstrating general applicability.
Furthermore, we proposed a new metric, the Normalized Power Spectrum
Similarity (NPSS), and demonstrated that it addresses and alleviates key
drawbacks of the commonly-used mean squared error in long-term motion
synthesis evaluation. Future research directions include incorporating the
NPSS metric into the parameter optimization process and developing models that
better train on multi-action data across all situations, particularly for
long-term motion synthesis.

## Acknowledgements

We gratefully acknowledge partial support from NSF grant CCF$\\#$1317560.

## References

  * [1] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [2] Ronan Boulic, Nadia Magnenat Thalmann, and Daniel Thalmann.  A global human walking model with real-time kinematic personification.  The visual computer, 6(6):344–358, 1990.
  * [3] Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  CoRR, abs/1412.3555, 2014.
  * [4] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik.  Recurrent network models for human dynamics.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4346–4354, 2015.
  * [5] Partha Ghosh, Jie Song, Emre Aksan, and Otmar Hilliges.  Learning human motion models for long-term predictions.  2017 International Conference on 3D Vision (3DV), pages 458–466, 2017.
  * [6] Kristen Grauman and Trevor Darrell.  Fast contour matching using approximate earth mover’s distance.  In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, volume 1, pages I–I. IEEE, 2004.
  * [7] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural Comput., 9(8):1735–1780, Nov. 1997.
  * [8] Alexander G. Ororbia II and Ankur Mali.  Biologically motivated algorithms for propagating local target representations.  CoRR, abs/1805.11703, 2018.
  * [9] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  IEEE Trans. Pattern Anal. Mach. Intell., 36(7):1325–1339, July 2014\.
  * [10] Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena.  Structural-rnn: Deep learning on spatio-temporal graphs.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5308–5317, 2016.
  * [11] Diederik P Kingma, Tim Salimans, and Max Welling.  Variational dropout and the local reparameterization trick.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2575–2583. Curran Associates, Inc., 2015.
  * [12] Lucas Kovar, Michael Gleicher, and Frédéric Pighin.  Motion graphs.  ACM Trans. Graph., 21(3):473–482, July 2002.
  * [13] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS’12, pages 1097–1105, USA, 2012\. Curran Associates Inc.
  * [14] Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio.  Professor forcing: A new algorithm for training recurrent networks.  In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 4601–4609. Curran Associates, Inc., 2016.
  * [15] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  nature, 521(7553):436, 2015.
  * [16] Sergey Levine, Jack M Wang, Alexis Haraux, Zoran Popović, and Vladlen Koltun.  Continuous character control with low-dimensional embeddings.  ACM Transactions on Graphics (TOG), 31(4):28, 2012.
  * [17] Julieta Martinez, Michael J Black, and Javier Romero.  On human motion prediction using recurrent neural networks.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4674–4683. IEEE, 2017.
  * [18] Alexander G Ororbia II, Tomas Mikolov, and David Reitter.  Learning simpler language models with the differential state framework.  Neural computation, 29(12):3327–3352, 2017.
  * [19] Vladimir Pavlovic, James M Rehg, and John MacCormick.  Learning switching linear models of human motion.  In Advances in neural information processing systems, pages 981–987, 2001.
  * [20] Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour.  Dropout improves recurrent neural networks for handwriting recognition.  In Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on, pages 285–290. IEEE, 2014.
  * [21] Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas.  The earth mover’s distance as a metric for image retrieval.  International journal of computer vision, 40(2):99–121, 2000.
  * [22] Iulian Vlad Serban, Alexander Ororbia II, Joelle Pineau, and Aaron Courville.  Piecewise latent variables for neural variational text processing.  In Proceedings of the 2nd Workshop on Structured Prediction for Natural Language Processing, pages 52–62, 2017.
  * [23] Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio.  A hierarchical latent variable encoder-decoder model for generating dialogues.  In AAAI, pages 3295–3301, 2017.
  * [24] Hedvig Sidenbladh, Michael J Black, and Leonid Sigal.  Implicit probabilistic models of human motion for synthesis and tracking.  In European conference on computer vision, pages 784–800. Springer, 2002.
  * [25] Yale Song, David Demirdjian, and Randall Davis.  Continuous body and hand gesture recognition for natural human-computer interaction.  ACM Trans. Interact. Intell. Syst., 2(1):5:1–5:28, Mar. 2012.
  * [26] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le.  Sequence to sequence learning with neural networks.  In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, pages 3104–3112, Cambridge, MA, USA, 2014. MIT Press.
  * [27] Graham W. Taylor, Geoffrey E Hinton, and Sam T. Roweis.  Modeling human motion using binary latent variables.  In B. Schölkopf, J. C. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19, pages 1345–1352. MIT Press, 2007.
  * [28] T. Tieleman and G. Hinton.  Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude.  COURSERA: Neural Networks for Machine Learning, 2012.
  * [29] Jack M Wang, David J Fleet, and Aaron Hertzmann.  Gaussian process dynamical models for human motion.  IEEE transactions on pattern analysis and machine intelligence, 30(2):283–298, 2008.
  * [30] Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals.  Recurrent neural network regularization.  CoRR, abs/1409.2329, 2014.
  * [31] Yi Zhou, Zimo Li, Shuangjiu Xiao, Chong He, Zeng Huang, and Hao Li.  Auto-conditioned recurrent networks for extended complex human motion synthesis.  In International Conference on Learning Representations, 2018.