[EN]

[EN]

# Simulation of magnetohydrodynamic flows of liquid metals with heat transfer
or magnetic stirring

Shashwat Bhattacharya111Corresponding author: e-mail shashwat.bhattacharya@tu-
ilmenau.de, phone +49 3677 69 2446 11 Seyed Loghman Sanjari 1122 Dmitry
Krasnov and Thomas Boeck 1111 [DE]Institute of Thermodynamics and Fluid
Mechanics, TU Ilmenau, P.O. Box 100565, 98684 Ilmenau  [DE]CTWe GmbH,
Kirchenstra$\upbeta$e, 91239 Henfenfeld

###### Abstract

We discuss the effects of nonhomogeneous magnetic fields in liquid metal flows
in two different configurations. In the first configuration, we briefly report
the impact of fringing magnetic fields in a turbulent Rayleigh-Bénard
convection setup, where it was shown that the global heat transport decreases
with an increase of fringe-width. The convective motion in regions of strong
magnetic fields is confined near the sidewalls. In the second configuration,
we numerically study the effects of an oscillating magnetic obstacle with
different frequencies of oscillation on liquid metal flow in a duct. The
Reynolds number is low such that the wake of the stationary magnetic obstacle
is steady. The transverse oscillation of the magnet creates a sinusoidal time-
dependent wake reminiscent of the vortex shedding behind solid obstacles. We
examine the behavior of the streamwise and spanwise components of the Lorentz
forces as well as the work done by the magnets on the fluid. The frequency of
the oscillation of the streamwise component of Lorentz force is twice that of
the spanwise component as in the case of lift and drag on solid cylindrical
obstacles. The total drag force and the energy transferred from the magnets to
the fluid show a non-monotonic dependence on the frequency of oscillation of
the magnetic obstacle indicative of a resonant excitation of the sinusoidal
vortex shedding.

##  1 Introduction

Magnetohydrodynamic (MHD) flows, i.e., flows of electrically conducting fluids
under the influence of magnetic fields, are frequently encountered in
engineering and astrophysical applications. In such flows, the fluid is acted
upon by the Lorentz force in addition to the force driving the flow.
Industrial and technological applications of such flows include heating,
pumping, stirring, and levitation of liquid metals, cooling blankets in fusion
reactors, and liquid-metal batteries. In the context of astrophysics, magnetic
fields strongly influence the flows in the sun and the stars and are
responsible for the formation of sunspots and solar flares.

Magnetoconvection has been studied extensively in the past, but most of the
studies focused on flows under the influence of a homogeneous magnetic field,
which is an idealized approximation. However, in most engineering and
astrophysical applications (such liquid metal batteries, cooling blankets in
fusion reactors, electromagnetic stirring, solar spots, etc.) the magnetic
fields are localized and thus vary in space [1, 2]. Further, strong
homogeneous fields in large regions of space can only be generated by magnets
of large size which are difficult to design and very costly to build and
operate [3]. Thus, it is important to understand the impact of spatially
varying magnetic fields on magnetohydrodynamic flows. Recently, Bhattacharya
et al. [4] studied the effects of spatially varying magnetic fields on MHD
flows driven by buoyancy (magnetoconvection); these effects will be briefly
summarized in this paper. There have been several studies on MHD duct flows
with different configurations of spatially varying fields (see, for example,
Sterl [5] and Prinz et al. [6]); however, in this paper, we focus specifically
on duct flows with a localized zone of applied magnetic field (henceforth
referred to as _magnetic obstacle_). Flows past stationary magnetic obstacles
have been studied before [7, 8, 9, 10, 11]. Similarities and differences
between stationary magnetic and solid obstacles has been discussed by Votyakov
and Kassinos[12]. Unsteady wakes were only found for fairly large Reynolds
numbers where the flow develops small-scale turbulent eddies [10]. In order to
realize an unsteady flow past a magnetic obstacle at a rather low Reynolds
number it seems necessary to add an additional periodic motion of the magnet.
We therefore consider the effects of _oscillating_ magnetic obstacles on MHD
duct flow in the present paper, which can be interesting in the context of
magnetic stirring. We also remark that oscillating solid obstacles have been
studied previously but it appears that such studies are lacking for magnetic
obstacles so far.

The outline of the paper is as follows. In Sec. 2, we discuss the mathematical
model. Section 3 describes the numerical method used in our simulations. We
discuss the results in Sec. 4 and conclude in Sec. 5.

##  2 Mathematical model

In this section, we describe the setup and the mathematical formulation of our
problems. The study will be conducted under the quasi-static approximation, in
which the induced magnetic field is neglected as it is very small compared to
the applied magnetic field. This approximation is fairly accurate for MHD
flows of liquid metals [2]. The governing equations of MHD flows are given by

$\displaystyle\nabla\cdot\boldsymbol{u}$ $\displaystyle=$ $\displaystyle 0,$
(1) $\displaystyle\frac{\partial\boldsymbol{u}}{\partial
t}+\boldsymbol{u}\cdot\nabla\boldsymbol{u}$ $\displaystyle=$
$\displaystyle-\nabla p+\nu\nabla^{2}\boldsymbol{u}+\boldsymbol{f},$ (2)

where $\boldsymbol{u}$ and $p$ are the velocity and pressure fields
respectively, $\nu$ is the kinematic viscosity, and $\boldsymbol{f}$ is the
total body force acting on the fluid. For MHD duct flow, $\boldsymbol{f}$ is
the specific Lorentz force (i.e. force per unit mass, henceforth denoted as
$\boldsymbol{f}_{L}$) and is given by

$\boldsymbol{f}=\boldsymbol{f}_{L}=\frac{1}{\rho}(\boldsymbol{j}\times\boldsymbol{B}_{0}),\quad\boldsymbol{j}=\sigma(-\nabla\phi+\boldsymbol{u}\times\boldsymbol{B}_{0}),\quad\nabla^{2}\phi=\nabla\cdot(\boldsymbol{u}\times\boldsymbol{B}_{0}).$
(4)

where $\boldsymbol{j}$ is the current density, $\boldsymbol{B}_{0}$ is the
imposed magnetic field strength, $\sigma$ and $\rho$ are the electric
conductivity and mean density of the fluid, respectively, and $\phi$ is the
electric potential.

In magnetoconvection, $\boldsymbol{f}=\boldsymbol{f}_{L}+\boldsymbol{f}_{b}$,
where $\boldsymbol{f}_{b}=\alpha gT\hat{z}$ is the buoyancy force, $\alpha$ is
the thermal expansion coefficient, $g$ is the gravitational acceleration, and
$T$ is the temperature field. Magnetoconvection is additionally governed by
the following thermal energy equation which describes the evolution of the
temperature field $T$:

$\frac{\partial T}{\partial t}+\boldsymbol{u}\cdot\nabla T=\kappa\nabla^{2}T,$
(5)

where $\kappa$ is the thermal diffusivity of the fluid.

MHD liquid-metal duct flows are governed by two nondimensional parameters: the
Reynolds number Re, which is the ratio of the inertial force to the viscous
force; and the Hartmann number Ha, which is the ratio of the Lorentz force to
the viscous force. Liquid-metal magnetoconvection is governed by three
nondimensional parameters: the Rayleigh number Ra, the ratio of the buoyancy
force to the dissipative forces; the Prandtl number Pr – the ratio of
kinematic viscosity to the thermal diffusivity; and the Hartmann number Ha.
These quantities are given by

$\mbox{{Re}}=\frac{UL}{\nu},\quad\mbox{{Ha}}=BL\sqrt{\frac{\sigma}{\rho\nu}},\quad\mbox{{Ra}}=\frac{\alpha
g\Delta L^{3}}{\nu\kappa},\quad\mbox{{Pr}}=\frac{\nu}{\kappa},$ (6)

where $U$, $L$, and $\Delta$ are the characteristic velocity, length, and
temperature scales respectively. For magnetoconvection, we consider the
Rayleigh-Bénard setup consisting of fluid enclosed between a cooler top plate
and a warmer bottom plate (the temperature difference between the plates being
$\Delta$), with the plates separated by a distance $H$. In this case, $H$ and
$\Delta$ respectively are the characteristic length and temperature scales for
Ha and Ra.

As discussed in Sec. 1, we describe the effects of spatially varying magnetic
fields in liquid metal flow for two configurations - (i) thermal convection in
a box, and (ii) pressure-driven duct flow. In the first configuration, we
consider a horizontally extended convection box of size $l_{x}\times
l_{y}\times H=16\times 32\times 1$ which is influenced by magnetic fields
generated by two semi-infinite permanent magnets. The north pole of one magnet
faces the bottom of the convection cell and the south pole of the second
magnet faces the top of the convection cell. These magnets extend from
$-\infty$ to $\infty$ in the $x$-direction, $l_{y}/2$ to $\infty$ in the
$y$-direction, from near the top wall to $\infty$ in the positive
$z$-direction, and from near the bottom wall to $-\infty$ in the negative $z$
direction. For a detailed description of the setup, the readers are referred
to Bhattacharya et. al [4]. In this configuration, the lateral component of
the magnetic field ($B_{x}$) vanishes, and the longitudinal and vertical
components respectively are logarithmic and inverse-tangent functions of the
spatial coordinates $y$ and $z$ and the gap $\delta$ between the magnetic
poles and the horizontal walls. The magnetic field distribution is such that
its strength is negligible for $0<y\lesssim l_{y}/2$, increases steeply at
$y\sim l_{y}/2$, and saturates close to its maximum value at $y\gtrsim
l_{y}/2$. When $\delta$ is increased keeping other parameters same, the total
magnetic flux through the convection cell remains the same, but the gradient
of the magnetic field at $y\sim l_{y}/2$ decreases, thereby increasing the
fringe-width of the magnetic field. The aim of the study was to determine the
effects of fringe-width on the heat and momentum transport.

The second configuration, which is the main focus of this paper, consists of
liquid metal flow in a duct with two oscillating permanent magnetic poles near
the top and bottom walls. The magnetic poles are semi-infinite in the
$z$-direction and measure $M_{x}=3$ units along the streamwise direction and
$M_{y}=4$ units along the spanwise direction in agreement with Votyakov and
Kassinos [12]. The spanwise dimension of the duct is $L_{y}=50$ units and the
height is $L_{z}=1$ unit. The vertical gap between the magnetic poles and the
liquid domain (one quarter of the layer height) also corresponds to Ref.[12].
A schematic diagram of the setup is shown in Fig. 1.

Figure 1: Schematic of the setup for the duct flow with a localized
oscillating magnetic field. The magnetic poles are semi-infinite in
$z$-direction and oscillate along $y$-direction. The height $L_{z}$ of the
duct is unity.

The magnetic field $\boldsymbol{B}=(B_{x},B_{y},B_{z})$ generated by the
magnetic poles is given by the formula derived by Votyakov et al. [13]. The
oscillation takes place along the spanwise direction and the $y$-coordinate of
the center of the magnet at time $t$ is given by

$y_{m}=A\sin(2{\pi}f_{0}t),$ (7)

where $A$ and $f_{0}$ are the amplitude and frequency of oscillation
respectively. The magnets therefore have a velocity $\boldsymbol{u}_{m}$ with
respect to the flow domain. Since the induction of currents depends on the
relative velocity between conductor and magnet, the difference
$\boldsymbol{u}-\boldsymbol{u}_{m}$ must be used in Ohm’s law (4b) and in the
charge conservation condition (4c).

In our work, the oscillation amplitude is set to $A=1$, i.e. the ratio
$A/M_{y}=0.25$. For the frequency we choose a reference value based on the
Strouhal number $\mbox{{St}}_{0}=0.25$ in Ref. [12]. The nondimensional
reference frequency in our work is therefore

$f_{s}=\frac{\mbox{{St}}_{0}U}{M_{y}}=\frac{0.25\times 1}{4}=0.0625$ (8)

where $U=1$ is the nondimensional mean streamwise velocity. The frequency
ratio is defined as $F=f/f_{s}$, i.e., the ratio of the frequency of
oscillation of the magnetic poles to that of vortex shedding for the
stationary magnetic obstacle of the same dimensions at a Reynolds number
$\mbox{{Re}}=900$ in Ref. [12].

##  3 Numerical method

We conduct direct numerical simulations of our setups using a second-order
finite difference code developed by Krasnov et al. [14, 15]. For the
magnetoconvection setup, a non-uniform grid of resolution $4800\times
9600\times 300$ was used. All the walls were rigid and electrically insulated
such that the electric current density $\boldsymbol{j}$ formed closed field
lines inside the cell. The top and bottom walls were fixed at $T=-0.5$ and
$T=0.5$ respectively, and the sidewalls were adiabatic. The Rayleigh number,
Prandtl number, and the Hartmann number based on the maximum value of the
vertical magnetic field were fixed at $\mbox{{Ra}}=10^{5}$,
$\mbox{{Pr}}=0.021$, and $\mbox{{Ha}}_{z,max}=120$. The gap $\delta$ between
the magnetic poles and the conducting plates was varied from $\delta=0.01H$ to
$\delta=9H$, where $H$ is the cell height.

For the configuration of flow past oscillating magnetic obstacle, we employ a
rectangular domain of dimensions $L_{x}\times L_{y}\times L_{z}=200\times
50\times 1$ with a grid resolution of $1024\times 384\times 32$. The fluid
enters the domain at $x=0$ with a nearly fully-developed laminar flow profile
that is approximated by the analytical expression

$u=\frac{\cosh\left(\frac{1.55L_{y}}{L_{z}}\left|\frac{2y}{L_{y}}\right|\right)-\cosh\left(\frac{1.55L_{y}}{L_{z}}\right)}{1-\cosh\left(\frac{1.55L_{y}}{L_{z}}\right)}\cdot\frac{\cosh\left(\frac{1.55L_{z}}{L_{y}}\left|\frac{2z}{L_{z}}\right|\right)-\cosh\left(\frac{1.55L_{z}}{L_{y}}\right)}{1-\cosh\left(\frac{1.55L_{z}}{L_{y}}\right)}.$
(9)

The fluid leaves the domain at $x=L_{x}$ where
$\partial\boldsymbol{u}/\partial x=0$. The magnetic poles are located $x=50$.
The mesh is non-uniform in $y$ and $z$-directions. The top, bottom, and
sidewalls are rigid (no-slip) and electrically insulated. We fix
$\mbox{{Re}}=400$ and the Hartmann number based on the maximum vertical
magnetic field as $\mbox{{Ha}}_{z,max}=70$, and vary the frequency ratio from
$F=0.2$ to $F=0.8$. It must be noted that the characteristic length and
velocity for the above nondimensional quantities are $L_{z}/2$ (that is, half
of the duct height) and the bulk horizontal velocity at the inlet ($U$),
respectively.

For both the simulation setups, the elliptic equations for pressure, electric
potential, and the temperature were solved based on applying cosine transforms
in along the directions with uniform grid-stretching and using a tridiagonal
solver along the direction with non-uniform grid stretching. The diffusive
term in the temperature transport equation is treated implicitly. The time
discretization of the momentum equation uses the fully explicit Adams-
Bashforth/Backward-Differentiation method of second order.

##  4 Results

In this section, we first briefly summarize the results on the
magnetoconvection simulations and then describe in detail the results on the
flow past oscillating magnetic obstacle.

###  4.1 Results on magnetoconvection

Figure 2:  (a) Schematic diagram of the magnetoconvection setup, and (b)
isosurface contours of vertical velocity $u_{z}=0.01$ (red) and $u_{z}=-0.01$
(blue) for magnetoconvection with $\delta/H=3$ [4]. The magnetic poles are
semi-infinite in $y$ and $z$-directions, and infinite along $x$-direction. The
fluid motion in the region of strong magnetic fields is restricted to narrow
zones adjacent to the sidewalls.

A schematic of the magnetoconvection setup is illustrated in Fig. 2(a). The
magnetic field generated by the magnets is strong enough to cease the flow in
high magnetic flux region of the convection cell. We observe that as the local
vertical magnetic field strength increases, the large scale structures become
thinner and align themselves perpendicular to the longitudinal sidewalls. The
dependence of the local Reynolds and Nusselt numbers on the local Hartmann
number (based on the vertical component of the magnetic field) was determined;
this dependence was observed to be independent of the fringe-width. The global
heat transport was observed to decrease with increasing fringe-width for
strong magnetic fields but decrease with increasing fringe-width for weak
magnetic fields. The convective motion became confined to the vicinity of the
sidewalls in the regions of strong magnetic fields as shown in Fig. 2(b). The
amplitudes of these wall modes were shown to exhibit a non-monotonic
dependence on the fringe-width.

For further details on the results, the readers are referred to Bhattacharya
et al. [4]. In the next section, we discuss the results for the MHD duct flow
setup.

###  4.2 Results on flow past oscillating magnetic obstacle

The simulations of the flow past magnetic obstacles are run for 300 convective
time units after reaching a fully-developed state. The contour plots of
instantaneous streamwise velocity are exhibited in Figs. 3(a–c) and those with
time-averaging in Figs. 3(d–f) for $F=0.2$, $F=0.5$, and $F=0.8$. The figures
show regions of reduced and even reversed streamwise velocity in the regions
of strong magnetic field and also in the wake of the magnetic obstacle. The
regions of reduced instantaneous velocity exhibit a wavy pattern. It can be
visually observed from Figs. 3(a–c) that as the magnets oscillate faster, the
wavelength of spatial oscillation decreases. There is an increase in the
amplitude of this path from $F=0.2$ to $F=0.5$, but the amplitude decreases
with a further rise in $F$. For $F=0.5$, the wake of the magnetic obstacle
comprises of small-scale eddies, indicating that the flow becomes turbulent.
The time-averaged streamwise velocity contours show that the length of the
reversed flow region first decreases as $F$ is increased to 0.5, and then
increases with a further increase of $F$.

We examine the components of the total Lorentz force in the streamwise
($f_{L,x}$) and spanwise ($f_{L,y}$) directions. Note that $f_{L,x}$ and
$f_{L,y}$ are the analog of the drag and lift forces, respectively, in flow
past solid cylinders. These quantites are

$f_{L,x}=\int_{-L_{z}/2}^{L_{z}/2}\int_{-L_{y}/2}^{L_{y}/2}\int_{0}^{L_{x}}(\boldsymbol{f}_{L}\cdot\boldsymbol{\hat{x}})\,dx\,dy\,dz,\quad
f_{L,y}=\int_{-L_{z}/2}^{L_{z}/2}\int_{-L_{y}/2}^{L_{y}/2}\int_{0}^{L_{x}}(\boldsymbol{f}_{L}\cdot\boldsymbol{\hat{y}})\,dx\,dy\,dz.$
(10)

Figures 4(a,b,c) exhibit the plots of the above quantities versus the
convective time $t$ for $F=0.5$, $F=0.6$, and $F=0.8$, respectively. The
figures show a periodic sinusoidal time dependence. The magnitude of $f_{L,x}$
is higher than $f_{L,y}$; however, $f_{L,y}$ oscillates with a higher
amplitude than $f_{L,x}$. The amplitude of oscillation increases with an
increase of $F$. The response frequency of $f_{L,y}$ is equal to the
excitation frequency $f_{0}$ of the magnets; however, the response frequency
of $f_{L,x}$ is twice of $f_{0}$. We further compute $\langle
f_{L,x}\rangle_{t}$, the streamwise component of Lorentz force averaged over
300 timeframes, and plot it versus the frequency ratio in Fig. 5(a). It can be
seen that $\langle f_{L,x}\rangle_{t}$ increases rapidly from $F=0.2$ to
$F=0.55$. On further increase of $F$, $\langle f_{L,x}\rangle_{t}$ decreases
sharply till $F=0.7$ above which $\langle f_{L,x}\rangle_{t}$ saturates close
to a constant value. Interestingly, the aforementioned behaviors of $f_{L,x}$
and $f_{L,y}$ closely resemble that of the drag and lift forces, respectively,
in flows past an oscillating cylinder [16, 17].

Figure 3: Contour plots of streamwise velocity $u_{x}$ for flows past
oscillating magnetic obstacle for different frequency ratios in the midplane
$z=0$. Instantaneous contour plots for (a) $F=0.2$, (b) $F=0.5$, and (c)
$F=0.8$. Time-averaged contour plots for (d) $F=0.2$, (e) $F=0.5$, and (f)
$F=0.8$.

For flows past an oscillating cylinder, the non-dimensional mechanical energy
transferred from the cylinder to the fluid is expressed as

$E=\frac{2}{\rho Ud^{2}}\int_{0}^{t_{P}}\frac{dy}{dt}f_{\mathrm{lift}}\,dt,$
(11)

where $t_{P}$ is the motion period, $d$ is the diameter of the cylinder, $y$
is the spanwise position of the cylinder’s axis, and $f_{\mathrm{lift}}$ is
the magnitude of the lift force [16]. In our work, the energy transferred from
the oscillating magnets to the fluid can be expressed similarly as follows:

$E=\int_{0}^{t_{P}}\frac{dy_{m}}{dt}\,f_{L,y}\,dt,$ (12)

where $t_{P}=$ 300 convective time units for our case. We compute $E$ for
different frequency ratios and plot it versus $F$ in Fig. 5(b). The figure
shows that $E$ is always positive, implying that the magnets perform work on
the fluid for all frequencies. The figure further shows that there is a
gradual growth of $E$ until $F=0.45$ and then it sharply decreases to a
minimum value at $F=0.53$. The energy transfer increases monotonically on
further increase of $F$. Interestingly, the point of minimum energy transfer
almost coincides with the point of maximum average streamwise Lorentz force.
This point corresponds to resonance where the velocity field exhibits stronger
fluctuations compared to other frequency ratios.

Figure 4: Time-series of the total spanwise and streamwise component of
Lorentz force ($f_{L,y}$ and $f_{L,x}$, respectively) in flow past oscillating
magnetic obstacles for the following frequency ratios: (a) $F=0.5$, (b)
$F=0.6$, and (c) $F=0.8$. The Lorentz force exhibits a sinusoidal time-
dependence.

Figure 5: For flow past oscillating magnetic obstacle: (a) Variation of the
time-averaged Lorentz force in the streamwise direction with the frequency
ratio, and (b) plots of energy transferred from the magnets to the fluid and
the phase angle between the spanwise Lorentz force and displacement of the
magnets. The above quantities exhibit a nonmonotonic dependence on $F$.

We finally examine the trends of the phase angle between the spanwise
component of Lorentz force and the spanwise displacement of the magnets. This
parameter is used as an indicator of energy transfer from the magnets to the
fluids [16] where a phase angle between 0 and 180 degrees indicates positive
energy transfer. We compute the phase angle using our data by fitting it with
a sinusoidal function using the method of least squares. The computed phase
angle is plotted versus the frequency ratio in Fig. 5(b). The figure shows
that the phase angle lies between 0 and 180 degrees, consistent with the fact
that the energy is transferred from the magnets to the fluid. The maximum
phase angle at $F=0.55$ reaches about 170 degrees.

##  5 Conclusions

In this paper, we numerically examined the effects of non-homogeneous magnetic
fields in liquid metal flows using a finite-difference fluid solver. We
briefly summarized the results of Bhattacharya et al. [4] in which the
influence of fringing magnetic fields on turbulent convection was studied. An
important finding was that for strong magnetic fields, the global heat
transport decreases with an increase of fringe-width, whereas for weak
magnetic fields, the heat transport marginally increases with an increase of
fringe-width. The convective motion gets confined near the sidewalls in
regions of strong magnetic fields.

We numerically examined the effects of oscillating magnetic obstacle with
different frequencies on liquid metal flow in a duct. We showed the presence
of reduced and reversed streamwise velocity in the regions of strong magnetic
field and in the wake of the magnetic obstacle. The regions of reduced
velocity exhibit a wavy pattern with the wavelength of spatial oscillation
decreasing with the excitation frequency of the magnets. The amplitude of wake
oscillation shows a non-monotonic dependence on the frequency of the magnets
and exhibits a maximum at a particular frequency $f_{\text{max}}$, which
appears to correspond to the point of maximum Lorentz force in the streamwise
direction and the minimum work done by the magnets on the fluid. The total
streamwise and spanwise components of the Lorentz force oscillate sinusoidally
with time with the same frequency and twice the frequency of the magnet’s
oscillation. The mean of the spanwise Lorentz force is zero. Its amplitude
increases with an increase of the frequency of oscillation of the magnets. The
frequency $f_{\text{max}}\approx 0.5f_{s}$ is considerably smaller than the
reference value $f_{s}$ taken from Ref. [12]. Although the stationary magnet
does not produce vortex shedding in our case, it seems plausible that our
value $f_{\text{max}}$ is indicative of the intrinsic shedding frequency when
Re (and possibly Ha) are increased further. Lower values than
$\mbox{{St}}_{0}=0.25$ of the Strouhal number of stationary magnetic obstacles
were also reported by Kenjereš et al. [10].

The authors are grateful to J. Schumacher for providing valuable contributions
to the study of convection under the influence of fringing magnetic fields. S.
Bhattacharya is supported by a postdoctoral fellowship of Alexander von
Humboldt Foundation, Germany.

## References

  * [1] P. A. Davidson, Annu. Rev. Fluid Mech. 31, 273–300 (1999).
  * [2] P. A. Davidson, An Introduction to Magnetohydrodynamics, second edition (Cambridge University Press, Cambridge, 2017).
  * [3] L. Barleon, K. J. Mack, and R. Stieglitz, The MEKKA-facility a Flexible Tool to Investigate MHD-flow Phenomena, Tech. Rep. FZKA 5821, 1996.
  * [4] S. Bhattacharya, T. Boeck, D. Krasnov, and J. Schumacher, J. Fluid Mech. 964, A31 (2023).
  * [5] A. Sterl, J. Fluid Mech. 216, 161–191 (1990).
  * [6] S. Prinz, V. Bandaru, Y. Kolesnikov, D. Krasnov, and T. Boeck, Phys. Rev. Fluids 1, 043601 (2016).
  * [7] S. Cuevas, S. Smolentsev, and M. A. Abdou, J. Fluid Mech. 553, 227–252 (2006).
  * [8] E. V. Votyakov, Y. Kolesnikov, O. Andreev, E. Zienicke, and A. Thess, Phys. Rev. Lett. 98, 144504 (2007).
  * [9] E. V. Votyakov, E. Zienicke, and Y. B. Kolesnikov, J. Fluid Mech. 610, 131–156 (2008).
  * [10] S. Kenjereš, S. ten Cate, and C. J. Voesenek, Int. J. Heat Fluid Flow 32, 510–528 (2011).
  * [11] S. Tympel, T. Boeck, and J. Schumacher, J. Fluid Mech. 735, 553–586 (2013).
  * [12] E. V. Votyakov and S. C. Kassinos, Phys. Fluids 21, 097102 (2009).
  * [13] E. V. Votyakov, S. C. Kassinos, and X. Albets-Chico, Theor. Comput. Fluid. Dyn 23, 571–578 (2009).
  * [14] D. Krasnov, O. Zikanov, and T. Boeck, Comput. Fluids 50, 46 – 59 (2011).
  * [15] D. Krasnov, A. Akhtari, O. Zikanov, and J. Schumacher, J. Comput. Phys. 474, 111784 (2023).
  * [16] H. M. Blackburn and R. D. Henderson, J. Fluid Mech. 385, 255–286 (1999).
  * [17] A. Placzek, J. F. Sigrist, and A. Hamdouni, Comput. Fluids 38, 80–100 (2009).


# A Unified Framework for Multimodal, Multi-Part Human Motion Synthesis

Zixiang Zhou
<EMAIL_ADDRESS>Yu Wan
<EMAIL_ADDRESS>Baoyuan Wang
<EMAIL_ADDRESS>
###### Abstract

The field has made significant progress in synthesizing realistic human motion
driven by various modalities. Yet, the need for different methods to animate
various body parts according to different control signals limits the
scalability of these techniques in practical scenarios. In this paper, we
introduce a cohesive and scalable approach that consolidates multimodal (text,
music, speech) and multi-part (hand, torso) human motion generation. Our
methodology unfolds in several steps: We begin by quantizing the motions of
diverse body parts into separate codebooks tailored to their respective
domains. Next, we harness the robust capabilities of pre-trained models to
transcode multimodal signals into a shared latent space. We then translate
these signals into discrete motion tokens by iteratively predicting subsequent
tokens to form a complete sequence. Finally, we reconstruct the continuous
actual motion from this tokenized sequence. Our method frames the multimodal
motion generation challenge as a token prediction task, drawing from
specialized codebooks based on the modality of the control signal. This
approach is inherently scalable, allowing for the easy integration of new
modalities. Extensive experiments demonstrated the effectiveness of our
design, emphasizing its potential for broad application. Website:
https://zixiangzhou916.github.io/UDE-2/

Figure 1: Our unified motion synthesis framework supports multimodal and
multi-part human motion generation. Left: text-to-motion. Middle: music-to-
motion. Right: speech-to-motion.

##  1 Introduction

Synthesizing realistic human motion is a pivotal component in many
applications, e.g., gaming, virtual reality, and has attracted rising
attention in both academia and industry. Recent progress in AI-Generated
Content (AIGC) [22, 23, 31, 37, 12, 32, 2, 38] paves a way to generate
realistic human motion from various modalities of control signals. These
approaches span across handling textual descriptions [22, 23, 31, 37], music
clips [32, 30], and speech segments [38, 8]. Despite the advancements, their
primary limitation is the focus on single-modality control signals,
overlooking the potential of multi-modal integration.

Among various approaches, UDE[37] can synthesize human motion from text or
music using one shared model. While its performance is promising, there are
several notable limitations: Firstly, it lacks support for speech as a control
signal. Secondly, it focuses solely on the torso, neglecting limb articulation
such as hand movements. Thirdly, it consolidates the motions of different
domains into one universal codebook — a design strategy we later argue is
suboptimal. Finally, updating the motion dataset for one domain adversely
impacts the other, compromising efficiency and scalability for further
enhancements.

The limitations of current methods become pronounced in practical settings
such as gaming or filmmaking, where motions may need to be generated from
textual cues, musical scores, or speech. A unified model that can handle these
diverse inputs to animate motion would greatly enhance efficiency and
practicality. It is equally critical for such a model to prioritize specific
body movements depending on the situation; for instance, torso dynamics are
crucial in dance sequences, while hand gestures are more prominent during
speech. Consequently, a unified approach that can tailor motion synthesis to
various body parts, guided by multimodal inputs, is highly valuable.

In this paper, we propose a unified multimodal-driven while-body motion
generation framework to overcome the above limitations. 1) We first learn body
part-specific and domain-specific VQ-VAEs to quantize motions of different
body parts and different domains into corresponding discrete token
representations. 2) To interpret the multimodal control signals effectively,
we harness the robust representational capabilities of various large-scale
pre-trained models, employing them as encoders for the respective signals.
These models generate embedding vectors as latent representations, which are
then projected to a uniform dimensionality for seamless integration. 3) To
learn the mapping from multimodal embedding representations to multi-part body
motions, a multimodal fusion transformer is first adopted. It transforms these
embeddings into a joint latent space. Subsequently, we frame the motion
synthesis as a language generation task, utilizing discrete codebooks for
various body parts and domains as different “vocabulary” in the GPT-like
language model[26]. Then we predict the next token from the corresponding
“vocabulary” given conditions and past predictions. 4) We decode the predicted
token sequences using corresponding decoders to get the final multi-part body
motions. To summarize, our significant contributions are outlined as follows:

  1. 1.

We present a pioneering unified human motion generation model that accepts
three distinct modalities: text, music, and speech, and enables the synthesis
of diverse body part motions, i.e., the torso and left/right hands.

  2. 2.

We design novel technical components within our model, including a two-stage
VQ-VAE tailored for torso motion quantization and a semantic enhancement loss,
both of which significantly elevate the effectiveness.

  3. 3.

We introduce a weight re-initialization for VQ-VAE training and a semantic-
aware sampling technique for diverse generations, to get a further performance
boost.

  4. 4.

Comprehensive evaluations of our method across various tasks – text-driven,
music-driven, and speech-driven motion generation – demonstrate its
competitive performance and robustness in these domains.

These contributions collectively represent a substantial leap forward in the
realm of multimodal human motion generation, showcasing the potential for more
nuanced and versatile applications.

##  2 Related Work

#### Motion Generation with Text

Recent advances in text-based motion generation have streamlined the process
from textual descriptions to motion sequences [23, 9, 31, 37, 34, 15, 24, 6,
35]. The major approaches normally employ VAE-based frameworks to encode text
into a latent space from which motions are then reconstructed. Concurrently,
diffusion models [31, 6] have been adopted, with conditional variants
denoising inputs into motion sequences informed by text. Additionally, the
influence of language models is evident, with research by [9, 37, 34, 15] and
contemporaries quantizing motions into discrete tokens and utilizing
autoregressive techniques for generation, predicting each subsequent motion
token based on text and prior tokens. Moreover, retrieval-based approaches
have also been explored [35, 24]. Here, a database of motions is queried using
text-encoded embeddings to find closely aligned motion sequences in the joint
latent space, effectively marrying text inputs with pre-existing motion data.

#### Motion Generation with Music

The synthesis of dance movements from music has gained significant interest,
employing a variety of techniques. Motion graph methods, as in [3] and [5]
segment dances and build graphs for transition between movements based on
dynamic and musical styles. These approaches yield high-fidelity dances but
necessitate pre-constructed motion graphs. Alternatively, direct generation
techniques like those developed by [14],[13], and [32] map music to motion
sequences using different architectures: transformer encoder-only, encoder-
decoder, and diffusion models, respectively, all focusing on continuous motion
prediction from musical inputs. On the other hand, discrete representation is
the choice for [30] and [37], where dance moves are not directly derived from
music. Instead, these models predict discrete token sequences conditioned on
the music, based on which regenerates the dance motion.

#### Motion Generation with Speech

In contrast to text- and music-driven synthesis, speech-driven motion
synthesis focuses more on gestural rather than torso movements, with
considerable research dedicated to addressing this challenge [38, 33, 4, 20,
25, 8, 1, 2]. Among these methods, diffusion models [38, 33, 1, 4] stand out
for their high diversity in generation, as seen in recent works. Yet, these
models often lack the capacity for precise editing or specified generation,
such as direct manipulation of latent embeddings. On the other hand,
variational generative approaches offer an alternative, as explored by [20,
25, 8, 2], where gesture attributes are modulated by distinct latent
embeddings produced by dedicated encoders. These models generate gesture
movements by integrating all control variables into a continuous
representation.

Figure 2: Overview. a: Hierarchical Motion Tokenization, where a two-stage
motion decoder is introduced to improve the reconstruction quality of torso
movements, compared with conventional VQ-VAE. b: Multimodal Multi-party Motion
Synthesis, including a T5[28] -like architecture for condition-to-motion
transform(b.1), a causal attention mechanism(b.2) and the fusion of an
auxiliary condition (b.3).

#### The Comparison with Our Work

As can be summarized, current human motion generation techniques tend to
specialize in single-modality control signals and fail to address multi-part
movements. To generate multi-part body movements conditioned on multimodal
control signals, different prior methods have to be engineered, making the
framework sophisticated and scalable. In contrast, we introduce a unified
framework capable of synthesizing diverse body part motions under the guidance
of multimodal inputs.

##  3 Method

The proposed method in Fig. 2 contains two major modules, 1) a hierarchical
motion tokenization module that quantizes whole-body motions to separate
codebooks, and 2) a multimodal motion generation module that synthesizes
whole-body motion tokens based on multimodal conditions.

###  3.1 Hierarchical Motion Tokenization

We decompose the whole-body pose representation into torso pose and
left/right-hand pose representation. Different encoding and decoding
strategies are adopted for different body parts.

#### Hand Motion Tokenization

Following the design of VQ-VAE in [37], we learn an encoder
$\mathcal{E}(\cdot)$ that transforms hand motion sequence
$x\in\mathbb{R}^{T\times c}$ to latent representations
$z\in\mathbb{R}^{T^{\prime}\times d}$, and quantize them by replacing the
nearest representation $z^{q}\in\mathbb{R}^{K\times d}$, where $z^{q}$ belongs
to learned discrete codebook
$\mathcal{Z}=\\{z_{k}^{q}|z_{k}^{q}\in\mathbb{R}^{K\times d}\\}$.

#### Hierarchical Torso Motion Tokenization

We propose a hierarchical VQ-VAE for torso pose quantization. We use SMPL[18]
rotation vector representation for torso poses, each of which is denoted as
$x_{i}=[t_{i},r_{i}]$, where $t_{i}$ is the root trajectory, and $r_{i}$ is
the rotation vectors. Instead of feeding $x_{i}=[t_{i},r_{i}]$ to encoder
directly, we use a modified representation: $\bar{x}_{i}=[\bar{t}_{i},r_{i}]$,
where $\bar{t}_{i}$ is calculated as Eq. 1:

$\bar{t}_{i}=\left\\{\begin{aligned} [0,0,0]&,&i=0,\\\
t_{i}-t_{i-1}&,&i>0.\end{aligned}\right.$ (1)

This modification focuses on pose and local trajectory, making the learned
discrete representation more robust to global trajectory shifting. We express
the encoding as: $z^{q}=\mathcal{E}(\bar{x})$, where
$z\in\mathbb{R}^{T^{\prime}\times d}$ is the encoded latent representation. We
follow the quantization process in [37] to obtain the quantized representation
$z^{q}\in\mathbb{R}^{K\times d}$.

During decoding, we first reconstruct local poses
$\bar{y}\in\mathbb{R}^{T\times c}$ as: $\bar{y}=\mathcal{D}_{l}(z^{q})$. For
each pose $\bar{y}_{i}$, it is expressed as:
$\bar{y}_{i}=[\bar{t}_{i},r_{i}]$. Since we can estimate $t_{i}$ from
$\bar{t}_{i}$ according to Eq.1, we can, hence, obtain sub-optimal global pose
as $\tilde{y}=\mathcal{R}(\bar{y})$, where $\mathcal{R}(\cdot)$ is the reverse
process of Eq.1. We refine $\tilde{y}$ using a 1D U-net network to obtain the
final pose: $y=\mathcal{D}_{g}(\tilde{y})$. The two-stage decoding could be
expressed as: $y=\mathcal{D}_{g}(\mathcal{R}(\mathcal{D}_{l}(z_{q})))$. The
loss of two-stage VQ-VAE is calculated as Eq. 2:

$\mathcal{L}_{VQ}=\mathcal{L}_{rec}+\beta_{1}\|sg[z]-z_{q}\|+\beta_{2}\|z-sg[z^{q}]\|$
(2)

where
$\mathcal{L}_{rec}=\alpha_{1}\mathcal{L}_{rec}^{l}+\alpha_{2}\mathcal{L}_{rec}^{\tilde{g}}+\alpha_{3}\mathcal{L}_{rec}^{g}$
is the reconstruction term that contains three sub-items, 1)
$\mathcal{L}_{rec}^{l}=\|\bar{x}-\bar{y}\|$ minimizes the distance between
local poses and reconstructed local poses, 2)
$\mathcal{L}_{rec}^{\tilde{g}}=\|x-\tilde{y}\|$ minimizes the distance between
global poses and reconstructed sub-optimal global poses, and 3)
$\mathcal{L}_{rec}^{g}=\|x-y\|$ minimizes the global poses and reconstructed
global poses. The definition of other two terms $\|sg[z]-z_{q}\|$,
$\|z-sg[z^{q}]\|$ follow [37].

#### Embedding Weights Re-initialization

We define the terminology “activated” as if the weight of token embedding is
updated during optimization, and it could be decoded back to a valid motion
segment. It is non-trivial to tune the factor $\beta_{1}$, $\beta_{2}$ in Eq.
2 to increase the activation rate. We propose a novel training technique,
called ”weight re-initialization” to solve this problem. For every $k$ step
during training, we count the activation rate of every token in the codebook
and sort them in descending order, the sorted tokens embeddings are:
$[e_{i}^{1},e_{j}^{2},...,e_{k}^{n}]$, where subscripts $i,j,k$ indicate their
original order, and superscripts indicate their sorted order. And the
activation rate after sorting follows: $r(e^{1})>r(e^{2})>...>r(e^{n})$. We
re-initialize the weight of $e^{k}$ as $e_{k}\leftarrow e^{n-k}+\delta$ if
$r(e^{k})<\tau$, where $\delta\sim\mathcal{N}(0,\sigma{I})$, and $\sigma$ is a
constant with very small value.

Figure 3: Pipeline of semantic enhancement. We propose a novel method to
enhance the degree of semantic distinction of condition embedding. We pre-
train a motion auto-encoder as prior and use it to enhance the degree of
semantic distinction of the hidden states of the encoder part. Figure 4:
Visual Comparison on Text-to-Motion Task. We evenly show 5 poses for each
motion to compare ours with SOTAs on text-to-motion tasks. The condition
language descriptions are shown in the very left column. Keywords are
highlighted.

Method | R Precision | FID $\downarrow$ | Div $\rightarrow$ | MM $\uparrow$
---|---|---|---|---
Top-1 $\uparrow$ | Top-2 $\uparrow$ | Top-3 $\uparrow$ | Top-4 $\uparrow$ | Top-5 $\uparrow$
GT | 63.49 | 78.50 | 84.66 | 88.14 | 90.35 | 0.082 | 14.45 | -
TM2T[9] | $42.27^{\pm{1.137}}$ | $57.04^{\pm{1.200}}$ | $65.10^{\pm{1.293}}$ | $70.58^{\pm{1.285}}$ | $74.82^{\pm{1.313}}$ | $10.80^{\pm{0.065}}$ | $13.77^{\pm{0.058}}$ | $1.62^{\pm{0.342}}$
MDM[31] | $48.88^{\pm{1.121}}$ | $65.53^{\pm{1.200}}$ | $72.81^{\pm{1.293}}$ | $78.12^{\pm{1.335}}$ | $81.47^{\pm{1.281}}$ | $4.53^{\pm{0.034}}$ | $13.88^{\pm{0.034}}$ | $7.12^{\pm{0.071}}$
T2M-GPT[34] | $51.36^{\pm{1.187}}$ | $64.85^{\pm{1.217}}$ | $75.88^{\pm{1.140}}$ | $78.75^{\pm{1.136}}$ | $82.32^{\pm{1.177}}$ | $6.67^{\pm{0.061}}$ | $13.65^{\pm{0.219}}$ | $5.65^{\pm{0.229}}$
MLD[6] | $48.85^{\pm{0.996}}$ | $65.52^{\pm{1.096}}$ | $73.21^{\pm{1.326}}$ | $78.52^{\pm{1.143}}$ | $80.97^{\pm{1.246}}$ | $11.85^{\pm{0.097}}$ | $13.64^{\pm{0.219}}$ | $4.63^{\pm{1.655}}$
UDE[37] | $46.69^{\pm 1.152}$ | $61.69^{\pm{1.259}}$ | $69.82^{\pm{1.239}}$ | $75.04^{\pm{1.240}}$ | $78.61^{\pm{1.236}}$ | $2.57^{\pm{0.019}}$ | $14.25^{\pm{0.034}}$ | $6.83^{\pm{0.121}}$
Ours | $54.48^{\pm{0.340}}$ | $68.02^{\pm{0.280}}$ | $74.50^{\pm{0.280}}$ | $78.62^{\pm{0.240}}$ | $81.63^{\pm{0.230}}$ | $2.54^{\pm{0.045}}$ | $14.72^{\pm{0.034}}$ | $8.48^{\pm{0.297}}$

Table 1: Results of text-to-motion. We compare the results of text-to-motion
generation between ours and the SOTA methods. Our method achieves better
semantic relevance, fidelity, and diversity performances.
$\hbox{\pagecolor{red!15}\rm Indicate best
results},\hbox{\pagecolor{yellow!15}\rm indicates second best results}$.
Figure 5: Visual Comparison on Music-to-Motion Task. To ease the
visualization, we modify the trajectory of each dance movement and present 5
poses per sequence.

Method | MDSC | FID $\downarrow$ | Div $\rightarrow$ | MM $\uparrow$ | BAS $\uparrow$
---|---|---|---|---|---
Acc. $\uparrow$ | Top-1 Retr. $\uparrow$ | I2I. $\downarrow$ | Simi. $\uparrow$
GT | 38.32 | 37.65 | 0.72 | 0.48 | 1.03 | 2.19 | - | 0.237
FACT[14] | 17.67 | 17.96 | 0.92 | 0.01 | 2.43 | 2.89 | - | 0.221
Bailando[30] | 37.04 | 38.01 | 0.74 | 0.32 | 1.04 | 2.12 | - | 0.233
UDE[37] | $20.69^{\pm{4.231}}$ | $19.84^{\pm{3.902}}$ | 0.89 | 0.17 | $0.38^{\pm{0.015}}$ | $2.04^{\pm{0.014}}$ | $1.85^{0.007}$ | $0.231^{\pm{0.019}}$
EDGE[32] | $23.93^{\pm{2.404}}$ | $23.69^{\pm{2.211}}$ | 0.86 | 0.41 | $1.17^{\pm{0.015}}$ | $1.85^{\pm{0.016}}$ | $1.68^{\pm{0.092}}$ | 0.260
Ours | $54.25^{\pm{4.564}}$ | $53.89^{\pm{4.119}}$ | 0.56 | 0.60 | $0.46^{\pm{0.023}}$ | $2.06^{\pm{0.021}}$ | $1.75^{\pm{0.028}}$ | $0.222^{\pm{0.025}}$

Table 2: Results of music-to-motion. We compare the style consistency, motion
quality, and diversity between ours and previous methods and the results
suggest that our method achieves the highest consistency.
$\hbox{\pagecolor{red!15}\rm Indicate best
results},\hbox{\pagecolor{yellow!15}\rm indicates second best results}$.

Method | ID Consis. | FID $\downarrow$ | Div $\rightarrow$ | MM $\uparrow$
---|---|---|---|---
Acc. $\uparrow$ | I2I. $\downarrow$
GT | 99.97 | 0.36 | 0.36 | 2.26 | -
DSG[33] | $99.91^{\pm{0.000}}$ | $0.44^{\pm{0.000}}$ | $0.55^{\pm{0.000}}$ | $2.18^{\pm{0.000}}$ | $1.80^{\pm{0.005}}$
Ours | $93.16^{\pm{7.900}}$ | $0.46^{\pm{0.051}}$ | $0.75^{\pm{0.054}}$ | $2.37^{\pm{0.018}}$ | $2.01^{\pm{0.006}}$

Table 3: Results of speech-to-motion. We evaluate the ID consistency, motion
quality, and diversity between our and SOTA methods.
$\hbox{\pagecolor{red!15}\rm Indicate best
results},\hbox{\pagecolor{yellow!15}\rm indicates second best results}$.
Figure 6: Visual Comparison on Speech-to-Motion Task. To ease the
visualization, we modify the trajectory of each dance movement and present 5
poses per sequence. We enlarge the display of hand motions and show them on
both sides of the whole-body motion display.

###  3.2 Multimodal Multi-Part Motion Generation

As illustrated in Fig. 2 (b.1), our method supports generating whole-
body(torso, left/right hand) motion with three different input modalities. We
describe the details below.

#### Multimodal Alignment

We leverage the power of large-scale pre-trained models as modality encoders.
We use CLIP text encoder[27] to understand and encode language descriptions.
We adopt MTR[7] to encode the music pieces, it is a multimodal model pre-
trained on text-music pair data with contrastive loss. We employ HuBERT[11], a
self-supervised trained model to learn speech embeddings. For all encoders, we
adopt the hidden states of their last layers as the latent embedding sequence.
We denote, for modality $p$, the input condition is
$x_{p}\in\mathbb{R}^{T_{p}\times c_{p}}$, the latent embedding is
$e_{p}\in\mathbb{R}^{T^{\prime}_{p}\times c^{\prime}_{p}}$. We introduce
adapter layers to align multimodal embeddings as:
$\mathcal{A}_{p}(e_{p}):\mathbb{R}^{T^{\prime}_{p}\times
c^{\prime}_{p}}\rightarrow\mathbb{R}^{T^{\prime}_{p}\times d}$. The multimodal
alignment could be expressed as:
$e_{p}=\mathcal{A}_{p}(\mathcal{E}_{m}(x_{p}))$.

#### Hierarchical Condition to Motion Tokens Transform

We adopt an encoder-decoder transformer architecture to transform multimodal
conditions to motion tokens. The encoder part takes as input the aligned
embeddings $e_{p}\in\mathbb{R}^{T^{\prime}_{p}\times d}$. We pad special
tokens to unify the lengths. We extract context information with full
attention mechanism and transform it to latent embedding
$\tilde{e}_{p}\in\mathbb{R}^{T^{\prime}_{p}\times d}$. Then we employ a
transformer decoder to map $\tilde{e}_{p}$ to motion tokens. We apply causal
masks to prevent future information from being leaked in training. We predict
the next token successively when generating. We propose a hierarchical decoder
architecture to generate tokens of different body parts. Assuming that torso
movements depend on the condition and past torso movements, the transform of
torso motion tokens is
$p_{\theta_{t}}(x_{t})=\prod_{i=1}^{n}p_{\theta_{t}}(x_{t,i}|x_{t,<i},C)$,
where $C$ is condition. Same assumption holds for hands movements as:
$p_{\theta_{h}}(x_{h})=\prod_{i=1}^{n}p_{\theta_{h}}(x_{h,i}|X_{h,<i},C)$. In
our hierarchical design, we adopt a base decoder to learn the common latent
information. We use separate head decoders to learn part-specific and
modality-specific motion tokens. Therefore, the hierarchical condition-to-
motion transform could be written as:

$p(x_{b,p})=\prod_{i=1}^{n}p_{\theta}(x_{b,p,i}|X_{b,p,<i},C_{p})p_{\theta_{b,p}}(x_{b,p,i}|X_{b,p,<i},C_{p})$
(3)

In Eq. 3, $b$ denotes body parts, and $p$ denotes modality of the condition,
$\theta$ denotes the parameters of base decoder, and $\theta_{b,p}$ denotes
the parameters of head decoders.

#### Separate Motion Vocabulary

We use separate vocabulary for different body parts subject to different
modalities. The use of separate vocabulary brings several advantages, 1) it
avoids sampling out-of-domain motion tokens, 2) updating the vocabulary of one
domain will not affect the others, and 3) it is efficient to introduce
additional modality to our pipeline as a plug-in.

#### Alignment of Auxiliary Condition

Our method supports taking any auxiliary conditions as a supplement. For
instance, when generating gesture motion from speech, the persona of
characters matters. We introduce trainable auxiliary vocabulary to the encoder
part as $\mathcal{Z}_{aux}=\\{e_{aux}|e_{aux}\in\mathbb{R}^{K\times d}\\}$,
where $K$ is the number of auxiliary conditions, and $d$ is the latent
dimension. We append the auxiliary embedding $e_{aux}$ to dominant condition
embeddings $e_{m}$ as: $e_{fused}=[e_{m},e_{aux}]$, shown in Fig. 2 (b.3).

#### Semantic Enhancement

Semantic meaning of condition signal plays a vital role in generating human
motion. It is self-explanatory that conditions with opposite semantic meanings
should correspond to human movements of obvious difference. For instance, the
human motions described by $C_{1}$: “a person is running forward” and
$C_{2}$:“a person is running backward” should present remarkable difference,
while the motion described by $C_{3}$:“a person is walking forward” is
expected to be more similar with the one described by $C_{1}$ than $C_{2}$. We
denote the motion generated by condition $C$ as $x_{c}$, the similarity
between two motion sequences and conditions are $s_{m}=\langle
x_{c_{1}},x_{c_{2}}\rangle$ and $s_{c}=\langle c_{1},c_{2}\rangle$. We
empirically found that for any two pairs of $\\{x_{c},c\\}$, $s_{m}\propto
s_{c}$ does not hold. It is particularly obvious for text-to-motion scenarios.
We propose a novel semantic enhancement loss to facilitate the alignment
between motion and condition. As shown in Fig. 3, we pre-train a motion auto-
encoder and use its encoder to encode motion as $e_{m}\in\mathbb{R}^{1\times
d}$. We denote $e_{c}\in\mathbb{R}^{1\times d}$ as temporal average of
condition embeddings $\tilde{e}_{p}\in\mathbb{R}^{T^{\prime}_{p}\times d}$
obtained by condition-to-motion encoder transformer. We align $e_{m}$ and
$e_{c}$ in latent space by maximizing the cross-modality similarity.
Specifically, we fix the motion encoder and optimize the condition-to-motion
encoder to align $e_{c}$ to $e_{m}$. This benefits learning motion-aware
representation from the condition. Our semantic enhancement loss is:

$\mathcal{L}_{sem}=\frac{1}{N}\Sigma_{i=1}^{N}(1-\langle
s_{m}^{i},s_{c}^{i}\rangle)$ (4)

#### Semantic-Aware Sampling

We propose a simple yet efficient sampling approach called “semantic-aware
sampling” to boost the semantic correlation while maintaining generation
diversity. We found that the token embedding similarity is proportional to
motion dynamic similarity. Instead of sampling a token from a probability
distribution, we sample one token using a different approach. Suppose token
$y_{i}$ corresponds to the largest probability
$p(x_{i}|x_{1},x_{2},...,x_{i}-1,C)$, and its embedding in codebook is
$z^{q}_{i}$. We calculate the pairwise distance between $z_{i}^{q}$ and
$\\{z_{j}^{q}\\}_{j=1}^{K}$ as:
$d_{ij}(z_{i}^{q},z_{j}^{q})=\|z_{i}^{q}-z_{j}^{q}\|_{2}$. Then convert the
distance to weight factors as:

$w_{ij}=\frac{exp(-d_{ij}/t)}{\Sigma_{j=1}^{K}{exp(-d_{ij}/t}})$ (5)

where $t$ is the temperature coefficient. Multiplying weight $w$ by
probability $p$ adjusts the distribution by reducing the probability that
dynamically irrelevant tokens have been sampled. Hence, increasing the
semantic consistency.

##  4 Experiments

Figure 7: Diverse Generated Motions. We generate 3 motion sequences from the
same condition to present that our method has promising ability of diverse
generation. Left: text-to-motion. Middle: music-to-motion. Right: speech-to-
motion.

Method | HumanML3D[9] | AIST++[14] | BEAT[17]
---|---|---|---
Two-stage | Re-init. | FID $\downarrow$ | MPJPE $\downarrow$ | MPJVE $\downarrow$ | MPJAE $\downarrow$ | FID $\downarrow$ | MPJPE $\downarrow$ | MPJVE $\downarrow$ | MPJAE $\downarrow$ | FID $\downarrow$ | MPJPE $\downarrow$ | MPJVE $\downarrow$ | MPJAE $\downarrow$
✓ | ✗ | 1.807 | 0.100 | 0.012 | 0.012 | 5.449 | 0.154 | 0.017 | 0.013 | 0.682 | 0.018 | 0.0018 | 0.0013
✗ | ✓ | 1.693 | 0.072 | 0.010 | 0.008 | 5.278 | 0.090 | 0.014 | 0.010 | 0.726 | 0.019 | 0.0018 | 0.0012
✓ | ✓ | 1.167 | 0.054 | 0.007 | 0.004 | 5.193 | 0.087 | 0.012 | 0.007 | 0.518 | 0.018 | 0.0017 | 0.0011

Table 4: Ablation Study on Variants VQ-VAE Designs. The results suggest two-
stage design and re-initialization strategy improve the performance well.
Indicate best results.

Method | Text-to-Motion | Music-to-Motion | Speech-to-Motion
---|---|---|---
Top-1 $\uparrow$ | Top-3 $\uparrow$ | FID $\downarrow$ | MM $\uparrow$ | Acc. $\uparrow$ | I2I. $\downarrow$ | FID $\downarrow$ | MM $\uparrow$ | Acc. $\uparrow$ | I2I. $\downarrow$ | FID $\downarrow$ | MM $\uparrow$
w/o SemE. | 53.99 | 74.22 | 2.57 | - | 54.14 | 0.56 | 0.44 | - | 94.48 | 0.50 | 0.66 | -
w/ SemE. | 55.06 | 75.60 | 2.56 | - | 54.29 | 0.55 | 0.60 | - | 93.92 | 0.49 | 0.83 | -
w/o SaS. | 52.71 | 73.06 | 2.48 | 9.02 | 53.85 | 0.57 | 0.53 | 1.86 | 92.61 | 0.49 | 0.89 | 2.02
w/ Sas. | 54.48 | 74.50 | 2.54 | 8.48 | 54.25 | 0.56 | 0.46 | 1.75 | 93.16 | 0.47 | 0.75 | 2.01

Table 5: Ablation Study on Semantic Enhancement and Semantic-aware Sampling.
We found semantic enhancement improves the semantic correlation and fidelity
for the text-to-motion domain. And semantic-aware sampling brings semantic
correlation gain with less fidelity and diversity loss on three domains.
Indicate best results. Figure 8: Hands Motion Generation. We evenly visualize
6 poses from a 10-second motion. The transcription is also shown at the top
row and keywords corresponding to selected frames are highlighted.

###  4.1 Datasets and Preprocessing

We utilize HumanML3D[10] and AIST++[14] for text-conditioned and music-
conditioned task respectively. For speech-conditioned tasks, we use BEAT[17],
a large-scale conversational gesture synthesis dataset. We follow [37] to
preprocess the HumanML3D and AIST++ datasets, resulting in motion sequences
with identical representation. For BEAT[17], we use the same motion
representation for the body sequences, and for left hand and right hand, we
use MANO[29] representation with 12 PCA components for each hand. Because BEAT
has a large amount of data, we only use data corresponding to 4 IDs for
training and evaluation(ID: 2, 4, 6, 8). For music and speech data, we use raw
acoustic wave sampled as 16KHz.

###  4.2 Implementation Details

For tokenization, the codebook size of body parts is set to 1024 for all
text-, music-, and speech-conditioned generations. The codebook size of hand
parts is 512. The embedding dimensions are 512 for all. We train 500 epochs
for VQ-VAEs. For condition-to-motion transform, the encoder part is an 8-layer
transformer encoder, the base decoder part is a 6-layer transformer decoder,
and the head decoder parts are a 2-layer transformer decoder, respectively. We
don’t train our model on multi-domain data jointly at first. We train it on
HumanML3D[10] for 200 epochs, then we integrate AIST++[14] to continue
training for 1000 epochs, then we add BEAT[17] for another 1000 epochs.

###  4.3 Evaluation Metrics

#### Text-to-Motion

We use Frechet Inception Distance(FID) to measure the distance between
generated motion and the real motion distributions, and diversity(Div) to
measure the feature distance among all generated motion sequences. We use
multimodality (MM) to measure the diversity of motions conditioned on the same
textual description. Finally, we measure the motion-text semantic correlation
using R-Precision[10]. Inspired by [24][19], we train a text-motion alignment
model for evaluating R-Precision. The detail of this model is described in
Appendix.

#### Music-to-Motion

We use FID, Div and MM for motion quality, diversity measurement. We use beat
alignment score(BAS)[14] to measure the alignment between motion and music
beat. In addition, we use MDSC[36] to measure the stylistic alignment between
generated dance movements and music pieces.

#### Speech-to-Motion

Similarly, we use FID for quality measurement, Div and MM for diversity
measurement. We observe that personality affects the motion patterns a lot in
speech-driven scenarios, it is beneficial if we can measure the ID
consistency. Inspired by [36], we proposed a metric to measure the alignment
between motion and ID. We describe the detail in Appendix.

###  4.4 Results

We evaluate our method on three tasks, 1) text-to-motion, 2) music-to-motion,
3) speech-to-motion, and compare our method with several SOTAs[23, 9, 31, 34,
37, 6] respectively. We discuss them in detail as follows.

#### Comparison on Text-to-Motion

For the text-to-motion task, We reproduce the SOTAs following their official
implementations using rotation vector as motion representation. We argue that
this setting eliminates the effect of different motion representations. The
comparison results are reported in Tab. 1, which suggests our method has a
better capability of mapping the semantic meaning of condition language
descriptions to motion sequences. Meanwhile, our method achieves better motion
quality and diversity compared with SOTAs. Fig. 4 shows a few examples for
visual comparison.

#### Comparison on Music-to-Motion

We reproduce [14, 30, 32, 37] to compare with ours. For every piece of
condition music, we generate dance sequences with the same length as input
music using our method and SOTAs. We crop the motion sequences to consecutive
5sec segments and evaluate the metrics on these segments. Tab. 2 shows that
our method has stronger ability in generating style-consistent dance movements
from music pieces than others while achieving competitive results in fidelity
and diversity. Fig. 5 also provides visual comparison results to facilitate
assessment.

#### Comparison on Speech-to-Motion

We compare ours with [33] on BEAT[17] using SMPL-X[21] representation. It
contains torso and left/right-hand motions. We generate long whole-body motion
sequences and crop to multiple 5sec segments for evaluation. We show the
results in Tab. 3 and the visual comparison in Fig. 6. Specifically, we show
in Fig. 8 that our method can synthesize vivid and smooth hand motions from
speech signals.

###  4.5 Ablation Study

For ablations, we validated the effectiveness of the two-stage VQ-VAE and the
weight re-initialization technique. Tab. 4 shows that applying both techniques
boosts the reconstruction quality of VQ-VAE on all three datasets.

We also investigate the effectiveness of two modules proposed in condition-to-
motion transform, 1) semantic enhancement(SemE.), and 2) semantic-aware
sampling(SaS.). The ablation results are presented in Tab. 5. We found that
SemS. consistently improves the semantic correlation and fidelity for the
text-to-motion domain, while this tendency is not obvious for music-
conditioned and speech-conditioned domains. We will discuss this in detail in
the Appendix. The SaS. brings semantic correlation gain with limited fidelity
and diversity loss on three domains.

##  5 Conclusion

In this paper, we propose a unified framework that supports synthesizing
whole-body human motions with up to 3 modalities of control signals. We adopt
separate codebooks and use a two-stage VQ-VAE and re-initialization strategy
to improve the motion quantization quality. In addition, we utilize an
encoder-decoder architecture to transform multimodal conditions into motion
tokens. Meanwhile, we propose a semantic enhancement module to advance the
semantic relevance and also introduce a semantic-aware sampling technique to
further boost the correlation with limited fidelity and diversity loss.
Extensive evaluation suggests our method achieves state-of-the-art in all
domains.

## References

  * Alexanderson et al. [2023] Simon Alexanderson, Rajmund Nagy, Jonas Beskow, and Gustav Eje Henter.  Listen, denoise, action! audio-driven motion synthesis with diffusion models.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–20, 2023.
  * Ao et al. [2022] Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu.  Rhythmic gesticulator: Rhythm-aware co-speech gesture synthesis with hierarchical neural embeddings.  _ACM Transactions on Graphics (TOG)_ , 41(6):1–19, 2022.
  * Au et al. [2022] Ho Yin Au, Jie Chen, Junkun Jiang, and Yike Guo.  Choreograph: Music-conditioned automatic dance choreography over a style and tempo consistent dynamic graph.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 3917–3925, 2022.
  * Chemburkar et al. [2023] Ankur Chemburkar, Shuhong Lu, and Andrew Feng.  Discrete diffusion for co-speech gesture synthesis.  In _GENEA: Generation and Evaluation of Non-verbal Behaviour for Embodied Agents Challenge 2023_ , 2023.
  * Chen et al. [2021] Kang Chen, Zhipeng Tan, Jin Lei, Song-Hai Zhang, Yuan-Chen Guo, Weidong Zhang, and Shi-Min Hu.  Choreomaster: choreography-oriented music-driven dance synthesis.  _ACM Transactions on Graphics (TOG)_ , 40(4):1–13, 2021.
  * Chen et al. [2023] Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu.  Executing your commands via motion diffusion in latent space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18000–18010, 2023.
  * Doh et al. [2023] SeungHeon Doh, Minz Won, Keunwoo Choi, and Juhan Nam.  Toward universal text-to-music retrieval.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023.
  * Ghorbani et al. [2023] Saeed Ghorbani, Ylva Ferstl, Daniel Holden, Nikolaus F Troje, and Marc-André Carbonneau.  Zeroeggs: Zero-shot example-based gesture generation from speech.  In _Computer Graphics Forum_ , pages 206–216. Wiley Online Library, 2023.
  * Guo et al. [2022a] Chuan Guo, Xinxin Xuo, Sen Wang, and Li Cheng.  Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts.  _arXiv preprint arXiv:2207.01696_ , 2022a.
  * Guo et al. [2022b] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng.  Generating diverse and natural 3d human motions from text.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5152–5161, 2022b.
  * Hsu et al. [2021] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460, 2021.
  * Huang et al. [2020] Ruozi Huang, Huang Hu, Wei Wu, Kei Sawada, Mi Zhang, and Daxin Jiang.  Dance revolution: Long-term dance generation with music via curriculum learning.  _arXiv preprint arXiv:2006.06119_ , 2020.
  * Li et al. [2022] Buyu Li, Yongchi Zhao, Shi Zhelun, and Lu Sheng.  Danceformer: Music conditioned 3d dance generation with parametric motion transformer.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 1272–1279, 2022.
  * Li et al. [2021] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa.  Ai choreographer: Music conditioned 3d dance generation with aist++.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 13401–13412, 2021.
  * Lin et al. [2023a] Junfan Lin, Jianlong Chang, Lingbo Liu, Guanbin Li, Liang Lin, Qi Tian, and Chang-wen Chen.  Being comes from not-being: Open-vocabulary text-to-motion generation with wordless training.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 23222–23231, 2023a.
  * Lin et al. [2023b] Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang.  Motion-x: A large-scale 3d expressive whole-body human motion dataset.  _Advances in Neural Information Processing Systems_ , 2023b.
  * Liu et al. [2022] Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng.  Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis.  In _European Conference on Computer Vision_ , pages 612–630. Springer, 2022.
  * Loper et al. [2015] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  SMPL: A skinned multi-person linear model.  _ACM Trans. Graphics (Proc. SIGGRAPH Asia)_ , 34(6):248:1–248:16, 2015.
  * Lu et al. [2023] Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung-Yeung Shum.  Humantomato: Text-aligned whole-body motion generation.  _arXiv preprint arXiv:2310.12978_ , 2023.
  * Pang et al. [2023] Kunkun Pang, Dafei Qin, Yingruo Fan, Julian Habekost, Takaaki Shiratori, Junichi Yamagishi, and Taku Komura.  Bodyformer: Semantics-guided 3d body gesture synthesis with transformer.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–12, 2023.
  * Pavlakos et al. [2019] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3d hands, face, and body from a single image.  In _Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Petrovich et al. [2021] Mathis Petrovich, Michael J Black, and Gül Varol.  Action-conditioned 3d human motion synthesis with transformer vae.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10985–10995, 2021.
  * Petrovich et al. [2022] Mathis Petrovich, Michael J Black, and Gül Varol.  Temos: Generating diverse human motions from textual descriptions.  _arXiv preprint arXiv:2204.14109_ , 2022.
  * Petrovich et al. [2023] Mathis Petrovich, Michael J Black, and Gül Varol.  Tmr: Text-to-motion retrieval using contrastive 3d human motion synthesis.  _arXiv preprint arXiv:2305.00976_ , 2023.
  * Qi et al. [2023] Xingqun Qi, Chen Liu, Lincheng Li, Jie Hou, Haoran Xin, and Xin Yu.  Emotiongesture: Audio-driven diverse emotional co-speech 3d gesture generation.  _arXiv preprint arXiv:2305.18891_ , 2023.
  * Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR, 2021.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67, 2020.
  * Romero et al. [2017] Javier Romero, Dimitrios Tzionas, and Michael J. Black.  Embodied hands: Modeling and capturing hands and bodies together.  _ACM Transactions on Graphics, (Proc. SIGGRAPH Asia)_ , 36(6), 2017.
  * Siyao et al. [2022] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu.  Bailando: 3d dance generation by actor-critic gpt with choreographic memory.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11050–11059, 2022.
  * Tevet et al. [2022] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano.  Human motion diffusion model.  _arXiv preprint arXiv:2209.14916_ , 2022.
  * Tseng et al. [2023] Jonathan Tseng, Rodrigo Castellon, and Karen Liu.  Edge: Editable dance generation from music.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 448–458, 2023.
  * Yang et al. [2023] Sicheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, Ming Cheng, and Long Xiao.  Diffusestylegesture: Stylized audio-driven co-speech gesture generation with diffusion models.  _arXiv preprint arXiv:2305.04919_ , 2023.
  * Zhang et al. [2023a] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen.  T2m-gpt: Generating human motion from textual descriptions with discrete representations.  _arXiv preprint arXiv:2301.06052_ , 2023a.
  * Zhang et al. [2023b] Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu.  Remodiffuse: Retrieval-augmented motion diffusion model.  _arXiv preprint arXiv:2304.01116_ , 2023b.
  * Zhou and Wang [2023a] Zixiang Zhou and Baoyuan Wang.  Mdsc: Towards evaluating the style consistency between music and dance.  _arXiv preprint arXiv:2309.01340_ , 2023a.
  * Zhou and Wang [2023b] Zixiang Zhou and Baoyuan Wang.  Ude: A unified driving engine for human motion generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5632–5641, 2023b.
  * Zhu et al. [2023] Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, and Lequan Yu.  Taming diffusion models for audio-driven co-speech gesture generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10544–10553, 2023.

## Supplementary Material

Figure 9: Pipeline of Text-Motion Alignment Model. (a) Measuring the cosine
similarity as text-motion alignment. (b) Pretrain motion auto-encoder. (c)
Train the text-motion alignment model using reconstruction and infoNCE loss.
Figure 10: Pipeline of Speech-to-Motion ID Consistency Model. (a) Measuring
the Intra2Inter(I2I.) and ID prediction accuracy(ID Acc.) as metrics. (b)
Pretrain motion auto-encoder. (c) Train the speech-to-motion ID consistency
model using clustering loss and cross entropy loss. Figure 11: Semantic
Enhancement on Text Embedding. We show the pairwise similarity of embeddings
obtained by: (a) pretrained text encoder, (b) multimodal condition encoder
trained w/o semantic enhancement, (c) multimodal condition encoder trained w/
semantic enhancement. (d) is the condition text descriptions. Figure 12:
Comparison of Activation Rate. We investigate the effectiveness of weights re-
initialization. We sort the activation rate in descending order. We use
logarithm coordinates for y-axis. Figure 13: More Results on Text-to-Motion
Generation. Figure 14: More Results on Music-to-Motion Generation. Figure 15:
More Results on Speech-to-Motion Generation. Figure 16: More Results on
Diverse Text-to-Motion Generation. Figure 17: More Results on Diverse Music-
to-Motion Generation. Figure 18: More Results on Diverse Speech-to-Motion
Generation.

##  Appendix A Text-Motion Alignment Model

Fig. 9 shows our approach in evaluating motion-to-text consistency. We encode
the motion sequence and text description to embeddings $z_{m}$ and $z_{t}$,
respectively. And measure the cosine similarity as the degree of alignment
between motion and text. We first train a motion auto-encoder and use its
encoder to encode the motion sequence. We adopt the model design proposed in
[22]. For the text encoder, we use the pretraind CLIP[27] text encoder to
encode the text description. As discussed previously(Section
$\color[rgb]{1,0,0}{3.2}$), the semantic discrepancy between motion and text
exists. We show how to train the text-motion alignment model in Fig. 9 (c). We
introduce two adapter layers, they transform the embeddings obtained by motion
and text encoder $z_{m}$ and $z_{t}$ to the same dimension. To align them in
semantically meaningful latent space, we regularize them through
reconstructing motion from aligned embeddings. During the training, we
initialize the encoders and decoders with their pretrained weights. We fix the
CLIP text encoder, and we tune the pretrained motion encoder and decoder with
a small learning rate. We train the adapter layers with a relatively large
learning rate. The alignment model is trained by optimizing the following
objective:

$\mathcal{L}=\mathcal{L}_{recon}+\mathcal{L}_{infoNCE}$ (6)

Where

$\mathcal{L}_{recon}=\|x-\tilde{x}\|_{2}$ (7)

is the reconstruction term, and

$\mathcal{L}_{infoNCE}=-\frac{1}{N}\Sigma_{i=1}^{K}(\log{\frac{exp(\langle
z_{m}^{i},z_{t}^{i}\rangle/\tau)}{exp(\Sigma_{j=1}^{K})(\langle
z_{m}^{i},z_{t}^{j}\rangle/\tau)}})$ (8)

is the infoNCE loss which helps align paired text-motion close while pushing
unpaired apart.

We compare different models and training designs as motion-text alignment
models in Tab. 6. It is noticeable that tuning pretrained $E_{m}$ and $D_{m}$
with a small learning rate brings remarkable performance improvement, which
proves the effectiveness of our design.

##  Appendix B Speech-to-Motion ID Consistency Evaluation Model

We argue that the characters of speakers affect the motion patterns severely
in terms of the speech-to-motion domain. We propose a novel method to evaluate
ID consistency. Inspired by [36], we formulate it as a clustering problem.
Specifically, if motions could be encoded to latent representation embeddings,
embeddings corresponding to the motions performed by the same person should be
close to each other in latent space, while those corresponding to different
persons should be away from each other. This is a typical clustering
hypothesis. Fig. 10 shows the proposed model. We propose two metrics to assess
ID consistency. 1) I2I: measures the ratio between intra-cluster to inter-
cluster distance. Given a target motion, its representation is encoded by
pretrained motion encoder as $z_{m}$. We calculate the intra-cluster distance
as: $d_{intra}=\|z_{m}-\hat{z}_{m}^{i}\|_{2}$, where $\hat{z}_{m}^{i}$ is the
learned cluster center of ID $i$, and the inter-cluster distance is calculated
as: $d_{inter}=\frac{1}{N-1}\Sigma_{j=1,j\neq
i}^{N}(\|z_{m}^{i}-\hat{z}_{m}^{j}\|_{2})$. Smaller I2I indicates higher ID
consistency. 2) Acc: measure the ID prediction accuracy. We first train a
motion auto-encoder, then we fix the encoder and a pretrained speech
encoder[11] as encoder priors. We use two adapter layers to align the speech
and motion embeddings in latent space. These adapter layers are only trainable
modules here, and we train them using clustering loss and cross-entropy
loss(Fig. 10 (c)).

##  Appendix C More Results of Ablation Study

Variants | R Top-1 $\uparrow$ | R Top-2 $\uparrow$ | R Top-3 $\uparrow$ | R Top-4 $\uparrow$ | R Top-5 $\uparrow$
---|---|---|---|---|---
tune $E_{m}$ | tune $D_{m}$
✗ | ✗ | 19.85 | 29.18 | 35.25 | 40.58 | 45.18
✗ | ✓ | 50.96 | 66.13 | 73.67 | 78.72 | 82.40
✓ | ✓ | 63.60 | 79.23 | 85.02 | 88.14 | 90.44

Table 6: Comparison on Text-Motion Alignment. We compare different models and
training designs on the text-motion alignment model. We found tuning $E_{m}$
and $D_{m}$ brings significant performance gain. Indicate best results.

#### Weights Re-Initialization

We compare the activation rate of codebook tokens with and without using the
weights re-initialization strategy on three datasets. Fig. 12 presents the
comparison results. We encode the test sets and count the activation rate. We
sort the results according to the activation rate in descending order and
display them in a histogram-like style. It is clearly observed that the
weights re-initialization strategy increases the code activation rate
remarkably.

#### Semantic Enhancement

We investigate the effectiveness of the proposed semantic enhancement module.
It is discussed previously that semantic enhancement is particularly effective
in the text-motion domain, while large-scale pre-trained models like CLIP[27]
are not able to capture the semantics accurately in some cases. For instance,
Fig. 11 (a) shows the semantic similarity between embeddings encoded by CLIP.
The associated textual descriptions are shown in Fig. 11 (d). As we can see,
CLIP text encoder fails to understand the key semantics. For example, ‘a
person walks forward quickly’ and ‘a person walks backward quickly’ have very
similar expressions, but their semantics vary a lot. But the similarity shows
that ‘a person walks forward quickly’ is closer to ‘a person walks backward
quickly’ than ‘a person is running forward quickly’. We believe that this is
because CLIP is trained on a pair of static images and text, and the dynamic
semantics in the text is not well learned as a result. We show in Fig. 11 (b),
(c) that our proposed semantic enhancement loss helps to learn better
semantics from textual description. For instance, it helps distinguish the
semantics from ‘forward’ and ‘backward’ accurately.

##  Appendix D More Results on Motion Generation

We present more motion generation results in Fig. 13, 14, 15. For the text-to-
motion domain, we show 16 motion sequences synthesized by different textual
descriptions. For the music-to-motion domain, we select 8 genres and show two
samples per genre. Each sample is a 4-second dance segment. For the speech-to-
motion domain, we show samples of the generation of 4 speakers. We display the
whole-body motions in the middle, and we enlarge the displays of left/right-
hand movements at the side for convenience of visualization.

##  Appendix E More Results of Diverse Generation

We show more results to demonstrate our method’s capability in synthesizing
diverse motion with high condition consistency in Fig. 16, 17, 18. We
synthesize 4 motion sequences from each condition signal to demonstrate the
diversity of generation. We show diverse torso movements for text-to-motion
and music-to-motion scenarios and present both torch and hand movements for
speech-to-motion scenarios. The results suggest that our method is able to
synthesize not only diverse torch movements but also diverse and smooth hand
movements.

##  Appendix F Limitation and Future Work

1) Due to the limitation of dataset[10, 14], our method is currently not
available for generating hand movements for text- and music-driven scenarios.
Collecting datasets with richer annotation(e.g. [16]), as well as enabling the
text- and music-conditioned hand motion synthesis is one of our prior future
works. 2) Our method supports synthesizing a single person’s motion at this
stage. Generating interactive actions conditioned on multimodal signals is of
great value. This includes interacting with other persons or with the
environment. Our future work also focuses on it. 3) Appropriate evaluation
metrics are beneficial to further improvements. Our future work also lies in
investigating better metrics.