# Axion Window on New Macroscopic Forces

Luca Di Luzio<EMAIL_ADDRESS>Istituto Nazionale di Fisica Nucleare,
Sezione di Padova, Via F. Marzolo 8, 35131 Padova, Italy Hector Gisbert
<EMAIL_ADDRESS>Dipartimento di Fisica e Astronomia ‘G. Galilei’,
Università di Padova,
Via F. Marzolo 8, 35131 Padova, Italy Istituto Nazionale di Fisica Nucleare,
Sezione di Padova, Via F. Marzolo 8, 35131 Padova, Italy Fabrizio Nesti
<EMAIL_ADDRESS>Dipartimento di Scienze Fisiche e Chimiche,
Università dell’Aquila, Via Vetoio, I-67100, L’Aquila, Italy Istituto
Nazionale di Fisica Nucleare, Laboratori Nazionali del Gran Sasso,
I-67100 Assergi (AQ), Italy Philip Sørensen<EMAIL_ADDRESS>Dipartimento di Fisica e Astronomia ‘G. Galilei’, Università di Padova,
Via F. Marzolo 8, 35131 Padova, Italy Istituto Nazionale di Fisica Nucleare,
Sezione di Padova, Via F. Marzolo 8, 35131 Padova, Italy

###### Abstract

Axion-mediated forces are enhanced by the presence of CP-violating axion
couplings, which are however tightly constrained by electric dipole moment
(EDM) searches. We discuss the underlying hypotheses behind different sources
of CP violation at high energies and the interplay between axion-mediated
force experiments and EDM observables. Specifically, we identify various
mechanisms, based on new sources of CP violation or Peccei-Quinn symmetry
breaking, that can significantly relax EDM constraints, leading to a
substantial redefinition of the QCD axion window for axion-mediated forces. By
considerably enlarging the QCD axion parameter space, our results provide a
well-motivated target for experiments probing scalar axion couplings to matter
fields. These include fifth-force tests of gravity, as well as searches for
spin-dependent forces via precision magnetometry, proton storage rings and
ultracold molecules.

Introduction. The Quantum Chromodynamics (QCD) axion originally emerged from
the need to “wash out” CP violation from strong interactions Peccei and Quinn
(1977a, b); Weinberg (1978); Wilczek (1978). This is achieved by introducing
the axion field $a$, endowed with a Peccei-Quinn (PQ) shift symmetry that
renders the QCD topological angle $\theta$ unphysical. Hence, the issue of CP
violation in strong interactions is traded for a dynamical question about the
axion vacuum expectation value (VEV), yielding an effective $\theta$ parameter

$\theta_{\rm eff}\equiv\frac{\left\langle a\right\rangle}{f_{a}}\,,$ (1)

where $f_{a}$ is the axion decay constant and the electric dipole moment (EDM)
of the neutron (nEDM) demands $\left|\theta_{\rm eff}\right|\lesssim 10^{-10}$
Abel  _et al._ (2020).

A general result Vafa and Witten (1984) ensures that the QCD ground state
energy density is minimal for $\theta_{\rm eff}=0$. However, one relies on the
fact that QCD is a vector-like theory and does not have extra sources of CP
violation beyond the $\theta$ term. Both conditions are violated in the
Standard Model (SM), which is a chiral theory that features a CP-violating
(CPV) phase in the quark sector. Indeed, in Ref. Georgi and Randall (1986) it
was estimated $\theta^{\rm SM}_{\rm eff}\sim 10^{-19}$ (see also Ellis and
Gaillard (1979); Khriplovich (1986); Gérard and Mertens (2012)). While this
irreducible SM contribution to $\theta_{\rm eff}$ turns out to be too tiny to
be testable, new CPV sources beyond the SM, generically expected to explain
the baryon asymmetry of the universe, might lead to a value of $\theta_{\rm
eff}$ closer to the nEDM bound.

An axion VEV could also be generated from ultraviolet (UV) sources of PQ
symmetry breaking. This is motivated by the fact that the U(1)PQ, as a global
symmetry, does not need to be exact and it is expected to be broken at least
by quantum gravity effects (see e.g. Kallosh  _et al._ (1995)).

Whatever its origin, a striking consequence of $\theta_{\rm eff}\neq 0$ is the
generation of a _scalar_ axion coupling to nucleons, $g^{S}_{aN}(\theta_{\rm
eff},\ldots)\propto\theta_{\rm eff}$ Moody and Wilczek (1984), where the
ellipses stand for other sources of CP violation, to be discussed in this
work. Including both scalar ($g^{S}_{af}$) and pseudo-scalar ($g^{P}_{af}$)
couplings to matter fields, with $f=p,n,e$ (cf. Eq. (2)), one obtains
different types of non-relativistic potentials mediated by the light axion
field, leading to new macroscopic forces, as suggested long ago by Moody and
Wilczek Moody and Wilczek (1984). Remarkably, scalar axion couplings, being
spin-independent, strongly enhance axion-mediated forces compared to the case
of pseudo-scalar couplings. An updated review of axion-mediated force
experiments and relevant limits can be found in Ref. O’Hare and Vitagliano
(2020) (see also Raffelt (2012); Irastorza and Redondo (2018); Sikivie
(2021)).

While phenomenological analyses often focus on a naive definition of the QCD
axion band, based on the simplifying assumption that $\theta_{\rm eff}$ is the
only source of CP violation Moody and Wilczek (1984), more recent studies have
started to systematically analyze different sources of CP violation and their
non-trivial interplay with EDM searches Pospelov (1998); Bertolini  _et al._
(2021); Okawa  _et al._ (2022); Dekens  _et al._ (2022); Plakkot  _et al._
(2023); Di Luzio  _et al._ (2023).

In this work, we provide a fresh look at the UV origin of the $g^{S}_{af}$
coupling and identify different mechanisms, based on new sources of CP or PQ
breaking, in order to maximize the scalar axion coupling relative to EDM
constraints, thus leading to a substantial redefinition of the traditional QCD
axion window for axion-mediated forces.

Axion couplings to matter. Including both CP-conserving and CPV couplings, the
axion effective Lagrangian with matter fields ($f=p,n,e$) reads

$\mathcal{L}_{af}=-g^{P}_{af}\,a\overline{f}i\gamma_{5}f-g^{S}_{af}\,a\overline{f}f\,,$
(2)

where $g^{P}_{af}=C_{af}m_{f}/f_{a}$ and the dimensionless couplings $C_{af}$
are model-dependent $\mathcal{O}(1)$ numbers. As a benchmark scenario, we
consider here the DFSZ model Zhitnitsky (1980); Dine  _et al._ (1981) at large
$\tan\beta$, yielding $C_{ap}=0.6$, $C_{an}=-0.3$ and $C_{ae}=1/3$ (see e.g.
Di Luzio  _et al._ (2020) for a derivation). The QCD axion mass and decay
constant follow the standard relation $m_{a}\simeq 5.7\,(10^{9}\,{\rm
GeV}/f_{a})\,{\rm eV}$.

In the traditional QCD axion scenario with a nonzero $\theta_{\rm eff}$, for
$g_{aN}^{S}$ $(N=p,n)$ one can use the standard isospin-symmetric formula of
Moody and Wilczek Moody and Wilczek (1984), with the correct extra 1/2 factor
Bertolini  _et al._ (2021), which gives at most

$\displaystyle g_{aN}^{S,\theta}$ $\displaystyle\simeq$ $\displaystyle 2\times
10^{-21}\left(\frac{10^{9}\,{\rm GeV}}{f_{a}}\right),$ (3)

assuming maximal $\theta_{\text{eff}}=1.2\times 10^{-10}$, saturating the
$d_{n}$ constraint. The value (3) defines the upper side of the traditional
QCD axion band for axion-mediated forces, currently employed for instance in
O’Hare (2020). The lower side of the band is estimated through the irreducible
SM contribution, via the CKM phase, as Okawa  _et al._ (2022)

$g_{aN}^{S,\text{CKM}}\simeq 10^{-30}\left(\frac{10^{9}\,{\rm
GeV}}{f_{a}}\right).$ (4)

Note, however, that a CPV scalar coupling to nucleons $g^{S}_{aN}$ can arise
from various mechanisms related to new sources of CP or PQ violation. These
can lead to $a)$ new terms in the meson plus axion chiral Lagrangian
($\chi$PT), that induce a shift of the axion and chiral vacuum leading to CPV
couplings Pich and de Rafael (1991), but also to $b)$ direct CPV axion-nucleon
terms in the baryon chiral Lagrangian (B$\chi$PT).

Experimental searches typically operate at the atomic level rather than
directly probing $g_{af}^{S}$. When considering an atomic system
${}^{A}_{Z}X$, the axion scalar coupling is effectively described by the
average

$\displaystyle
g^{S}_{aX}\equiv\frac{A-Z}{A}\,g_{an}^{S}\,+\frac{Z}{A}\,g_{ap}^{S}\,+\frac{Z}{A}\,g_{ae}^{S}\,.$
(5)

In our theoretical predictions, we will consider the case of Tungsten
$(A=184,Z=74)$ which pertains to ARIADNE Arvanitaki and Geraci (2014); Geraci
_et al._ (2018). Other elements, relevant for fifth-force searches, imply a
relative variation from the Tungsten benchmark that is below the $10\%$ level.

In the following, we explore the possibility of maximizing this scalar axion-
matter coupling in comparison to $g_{aX}^{S,\theta}$, equal to the traditional
$g_{aN}^{S,\theta}$ in the isospin limit, while satisfying the EDM
constraints.

CPV couplings from $\boldsymbol{\chi}$PT. In the $\chi$PT Lagrangian, the
terms generated by CP or PQ breaking lead to a realignment of the vacuum,
namely a nonzero $\theta_{\rm eff}$ together with $\pi_{0}$, $\eta_{8}$ and
$\eta_{0}$ meson VEVs if chiral symmetry is also broken. These four vacuum
shifts induce CPV baryon couplings in the B$\chi$PT Lagrangian, reported for
completeness in Appendix A. It is important to note that the induced couplings
generally depend on just _three_ combinations. Indeed, the B$\chi$PT
Lagrangian feels the vacuum realignment only via the three quark masses, the
only spurions of chiral $SU(3)_{V}\times U(3)_{A}$. As a result, all effects
depend on the following three chiral phases (with $q=u,d,s$):

$\displaystyle\\!\\!\\!\\!\\!\alpha_{q}\\!$ $\displaystyle=$
$\displaystyle\\!\\!\left[\frac{\lambda_{3}}{2}\frac{\left\langle\pi_{0}\right\rangle}{F_{\pi}}+\\!\frac{\lambda_{8}}{2}\frac{\langle\eta_{8}\rangle}{F_{\pi}}+\\!\frac{\mathbf{1}}{\sqrt{6}}\frac{\left\langle\eta_{0}\right\rangle}{F_{\pi}}\right]_{qq}\\!\\!+\frac{m_{*}\theta_{\text{eff}}}{2\,m_{q}},\
$ (6)

where the $\lambda$’s are the Gell-Mann matrices,
$m_{*}=(m_{u}^{-1}+m_{d}^{-1}+m_{s}^{-1})^{-1}$, and the pion decay constant
is $F_{\pi}\simeq 92$ MeV. For the neutron and proton CPV couplings, one can
rewrite the results of Bertolini  _et al._ (2021) (see also Cirigliano  _et
al._ (2017); Bertolini  _et al._ (2020)) in a simple and general way in terms
of $\alpha$’s:

$\displaystyle g^{S}_{ap,\,n}\\!$
$\displaystyle=\\!-\frac{8m_{*}B_{0}}{f_{a}}\bigg{(}b_{0}{\normalsize\sum_{q}}\alpha_{q}+b_{+}\alpha_{u,d}+b_{-}\alpha_{s}\bigg{)}$
(7) $\displaystyle\simeq 10^{-11}\frac{10^{9}\,{\rm
GeV}}{f_{a}}(2.4\alpha_{u,d}+2.8\alpha_{d,u}+1.5\alpha_{s})\,,$

with 20% hadronic uncertainties, see Appendix A for details. Expressions in
terms of $\alpha$’s can be derived also for nucleon-meson CPV couplings (cf.
Appendix A) so that this fact holds in general even for EDM observables. We
thus find that from $\chi$PT there is a model-independent correlation between
EDMs and $g^{S}_{ap,n}$, all driven by three phases $\alpha_{u,d,s}$.

EDM constraints on CPV couplings. We focus first on the CPV effects induced in
$\chi$PT, considering the $g_{an,p}^{S}$ terms in Eq. (5), with $g_{ae}^{S}$
set to zero. The strongest EDM limits are currently imposed by neutron Abel
_et al._ (2020) and Mercury Graner  _et al._ (2016). For them, we find the
constraints

$\displaystyle\frac{d_{n}}{d_{n}^{<}}$
$\displaystyle\,\simeq\,10^{10}|3.7\,\alpha_{u}+5.9\,\alpha_{d}+40.0\,\alpha_{s}|\lesssim
1\,,$ (8) $\displaystyle\frac{d_{\text{Hg}}}{d_{\text{Hg}}^{<}}$
$\displaystyle\,\simeq\,10^{10}|3.1\,\alpha_{u}+5.0\,\alpha_{d}+36.0\,\alpha_{s}|\lesssim
1\,,\quad$ (9)

with 10% and 50% hadronic uncertainties respectively, see Appendix A for
details.

While other nuclei lead to weaker bounds, more constraints arise from EDMs of
paramagnetic systems from the contribution of two-photon exchange processes
between electrons and the nucleus induced by CP-odd semileptonic interactions
Flambaum  _et al._ (2020). The most stringent one, set by paramagnetic ThO
Andreev  _et al._ (2018), leads to the additional constraint (cf. Appendix A)

$\displaystyle 10^{10}|0.11\,\alpha_{u}-0.22\,\alpha_{d}|\lesssim 1\,,$ (10)

with 10% hadronic uncertainties, see Appendix A for details. The limit in Eq.
(10) is slightly weaker, but together with Eqs. (8) and (9), constrain all
$\alpha$’s to be of the order of $10^{-9\div 10}$. This implies that the axion
scalar coupling in Eq. (7) cannot deviate much from the traditional value.
Including uncertainties, we find at most
$|g^{S}_{aX}/g^{S,\theta}_{aX}|\lesssim\mathcal{O}(10)$.

In conclusion, the three EDM constraints fix the three phases, impeding a
relevant enhancement of $g_{aX}^{S}$ relative to the traditional QCD axion
case $g_{aX}^{S,\theta}$. This result is model-independent, as far as
$\chi$PT-induced effects are concerned.

The situation could potentially change with direct contributions to the
B$\chi$PT Lagrangian. New terms in general lead to additional combinations
different from those in Eq. (6), possibly evading the EDM limits. The possible
new operators, classified in Dekens  _et al._ (2022), suffer from
uncertainties in the low-energy constants (LECs), not determined by chiral
symmetry, and a model-independent analysis is premature. One can, of course,
imagine that more operators conspire to bypass the EDM constraints while
allowing for a larger $g_{aX}^{S}$. This scenario, hard to obtain in
predictive theories, could be checked in specific models.

To this aim, theoretical work on the LECs would be most welcome, while on the
experimental side future improvements on EDMs such as Radon Bishof  _et al._
(2016) will also be important to sharpen the connection between EDMs and
$g_{aN}^{S}$.

Semi-leptonic operators. We turn here to CPV effects induced only by the
scalar axion-electron coupling $g_{ae}^{S}$, the third term in Eq. (5). This
coupling can be generated by semileptonic operators as

$\mathcal{L}_{\text{sl}}=C^{P}_{eq}\,(\overline{e}_{L}e_{R})(\overline{q}\gamma_{5}q)+\text{h.c.}\,,$
(11)

with implicit sum over $q=u,d,s$. After the canonical axion-dependent chiral
rotation of the quarks (cf. Appendix A), Eq. (11) leads to Dekens  _et al._
(2022)

$g_{ae}^{S}=\frac{m_{*}\,B_{0}\,F_{\pi}^{2}}{2\,f_{a}}\,{\rm
Im}\sum_{q}\frac{1}{m_{q}}C^{P}_{eq}\,.$ (12)

The interactions in Eq. (11) also generate 4-fermion operators involving
nucleons and electrons Dekens  _et al._ (2022)

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\mathcal{L}_{eN}$
$\displaystyle=$
$\displaystyle-\frac{G_{F}}{\sqrt{2}}\bigg{\\{}i\,\overline{e}\gamma_{5}e\,\overline{N}\left(C_{S}^{(0)}+\tau_{3}C_{S}^{(1)}\right)N$
(13) $\displaystyle+$
$\displaystyle\overline{e}e\,\frac{\partial_{\mu}}{m_{N}}\left[\overline{N}\left(C_{P}^{(0)}+\tau_{3}C_{P}^{(1)}\right)S^{\mu}N\right]\bigg{\\}}\,,$

where $N=(p\,\,n)^{T}$ is the nucleon doublet, $S^{\mu}$ is its spin, $G_{F}$
is the Fermi constant, and $C_{S,P}^{(0,1)}$ are the Wilson coefficients
encoding the short-distance coefficients $C^{A}_{eq}$ of Eq. (11). The Mercury
EDM or CPV effects induced in polar molecules like ThO Andreev  _et al._
(2018), YbF Hudson  _et al._ (2011), and HfF Cairncross  _et al._ (2017)
receive contributions from $C_{S,P}^{(0,1)}$. Using Eq. (25) in Ref. Dekens
_et al._ (2022), we find the conditions on the $C^{P}_{eq}$ to cancel them:

$C^{P}_{eu}=C^{P}_{ed}=C^{P}_{es}\equiv C\,,$ (14)

for some $C$. These conditions still allow for a non-zero $g_{ae}^{S}$ from
Eq. (12):

$\displaystyle g_{ae}^{S}$
$\displaystyle=-\frac{B_{0}\,F_{\pi}^{2}}{2\,f_{a}}{\rm Im}\,C\,.$ (15)

Hence, one can evade the EDM limits by construction. The coefficient
$\text{Im}\,C$ is constrained by the high-$p_{T}$ tails of the $pp\to\ell\ell$
Drell-Yan processes. The results in Ref. Allwicher  _et al._ (2023) imply
$\text{Im}\,C\lesssim(2\,\text{TeV})^{-2}$, which allows for a substantial
enhancement of the scalar axion coupling,
$|g_{aX}^{S}/g_{aX}^{S,\theta}|\lesssim 1\times 10^{3}$.

PQ-breaking as the origin of CPV couplings. Another possibility to generate a
scalar axion coupling to fermions is to consider a source of PQ breaking,
which also breaks CP in the axion sector. A common choice is provided by
operators of the type $\mathcal{L}_{\rm\cancel{\rm
PQ}}\supset-e^{i\delta}\phi^{n}\Lambda^{-n+4}$, with $\delta$ a generic phase,
$\Lambda$ is the UV scale at which the operator is generated, and
$\phi=\frac{f_{a}}{\sqrt{2}}e^{ia/f_{a}}$ is a complex scalar where the axion
is the angular mode and the radial mode is integrated out. Including the QCD
axion potential, $V_{\rm QCD}=-\chi_{\rm QCD}\cos\frac{a}{f_{a}}$ with
$\chi_{\rm QCD}\approx(76\,\text{MeV})^{4}$, the induced axion VEV reads
$\theta_{\rm
eff}\simeq-2^{1-\frac{n}{2}}n\Lambda^{4-n}f_{a}^{n}\sin\delta/\chi_{\rm QCD}$.
Note that such a PQ-breaking scenario directly matches to the traditional QCD
axion coupling in Eq. (3).

Another class of operators, recently discussed in Refs. Zhang (2023, 2024) as
a solution to the axion domain wall problem Sikivie (1982), takes the generic
form $(\phi/\Lambda)^{n}\mathcal{O}_{\rm SM}$, where $\mathcal{O}_{\rm SM}$ is
an operator made of SM fields. To maximize the contribution to $g^{S}_{aX}$ in
Eq. (5), relative to EDM bounds, we consider a $\phi^{n}$ coupling to the
electron Yukawa,111Similar operators with $\phi^{n}$ coupled to light quarks
or gluons yield an unsuppressed long-distance contribution to $\theta_{\rm
eff}$ Zhang (2023), preventing a significant contribution to $g^{S}_{aX}$.

$\mathcal{L}_{\rm\cancel{\rm
PQ}}\supset-e^{i\delta}\left(\frac{\phi}{\Lambda}\right)^{\\!\\!n}\\!\\!\frac{\sqrt{2}m_{e}}{v}\overline{L}_{L}He_{R}+\text{h.c.}\,,$
(16)

where $v=246$ GeV. This operator generates a scalar axion-electron coupling
directly at tree level

$\displaystyle
g^{S}_{ae}=n\left(\frac{f_{a}}{\sqrt{2}\Lambda}\right)^{n}\frac{m_{e}}{f_{a}}\sin\delta\,,$
(17)

and, after setting $\left\langle H\right\rangle=(0\ \ v/\sqrt{2})^{T}$, it
also leads to an axion tadpole, $V_{\rm\cancel{\rm PQ}}=-\sigma a+\ldots$, at
one loop. In the leading-log approximation, we find

$\sigma(\mu)=\frac{n}{2\pi^{2}}\left(\frac{f_{a}}{\sqrt{2}\Lambda}\right)^{\\!n}\frac{m_{e}^{4}}{f_{a}}\sin\delta\ln\left(\frac{v}{\mu}\right),$
(18)

where $\mu$ denotes the renormalization scale. Including the contribution of
the QCD axion potential, we obtain the induced axion VEV

$\displaystyle\theta_{\rm eff}$
$\displaystyle\simeq\frac{n}{2\pi^{2}}\left(\frac{f_{a}}{\sqrt{2}\Lambda}\right)^{\\!n}\\!\frac{m_{e}^{4}}{\chi_{\rm
QCD}}\sin\delta\,\ln\left(\frac{v}{1\,\text{GeV}}\right)\,,$ (19)

where we took $\mu=1$ GeV to assess the nEDM bound. By using Eq. (19) to
express Eq. (17) in terms of $\theta_{\rm eff}$, we obtain

$g^{S}_{ae}=\frac{2\pi^{2}}{\ln\left(\frac{v}{1\,\text{GeV}}\right)}\frac{\chi_{\rm
QCD}}{m_{e}^{3}f_{a}}\theta_{\rm eff}\,.$ (20)

The nEDM bound, $|\theta_{\rm eff}|\lesssim 10^{-10}$, provides the main
limiting factor for this coupling,222Other direct contributions to EDMs come
from axion exchange involving $aG\tilde{G}$ and $a\overline{e}e$ vertices,
leading to semileptonic operators as in Eq. (13). Using results from Di Luzio
_et al._ (2021a), we obtain $g^{S}_{ae}\lesssim
2.2\,(f_{a}/10^{9}\,\text{GeV})$ from ThO EDM, which is negligible compared to
the bound from nEDM in Eq. (20). which in this scenario dominates by far the
atomic average in Eq. (5), allowing for
$|g_{aX}^{S}/g_{aX}^{S,\theta}|\lesssim 2\times 10^{7}$.

$Z_{{\cal N}}$ axion. The last possibility that we consider is a modification
of the standard $m_{a}$–$f_{a}$ QCD relation, suppressing $m_{a}$ for fixed
$f_{a}$. This can be achieved by employing ${\cal N}$ mirror copies of the SM
Hook (2018); Di Luzio  _et al._ (2021b, c), with ${\cal N}$ odd and
$\text{SM}_{k}\to\text{SM}_{k+1(\text{mod}\,{\cal N})}$ under the $Z_{{\cal
N}}$ symmetry and the axion acting non-linearly: $a\to a+2\pi k/{\cal N}$,
with $k=0,\ldots,{\cal N}-1$. In this way, the axion potential gets
exponentially suppressed and, in the large ${\cal N}$ limit, the axion mass
relative to the standard QCD axion mass scales as Di Luzio  _et al._ (2021b)

$\frac{(m_{a})_{{\cal
N}}^{2}}{m_{a}^{2}}\simeq\frac{\sqrt{1-z^{2}}(1+z)}{\sqrt{\pi}}{\cal
N}^{3/2}z^{{\cal N}-1}\,,$ (21)

where $z=m_{u}/m_{d}\simeq 0.48$. In these scenarios, an important constraint
on $f_{a}$ arises from the fact that the exponential axion mass suppression is
spoiled by finite density effects in stellar environments Hook and Huang
(2018). In particular, the strongest constraints arise from the modifications
of the mass-radius relationship of white dwarfs, which exclude $33\leq{\cal
N}\leq 69$ Balkin  _et al._ (2024). Assuming ${\cal N}\leq 31$, we find
$|g_{aX}^{S}/g_{aX}^{S,\theta}|\lesssim 5\times 10^{3}$.

Axion-mediated force experiments. Depending on the combination of couplings
involved, axion-mediated non-relativistic potentials can be of three types:
$g^{S}_{af}g^{S}_{af}$ (monopole-monopole), $g^{S}_{af}g^{P}_{af}$ (monopole-
dipole), or $g^{P}_{af}g^{P}_{af}$ (dipole-dipole). The idea of searching for
dipole-dipole axion interactions in atomic physics is as old as the axion
itself Weinberg (1978). However, dipole-dipole forces turn out to be spin-
suppressed and suffer from large backgrounds from ordinary magnetic forces.
Furthermore, our models enhance only scalar couplings and thus do not improve
the situation for dipole-dipole interactions. Hence, in the following, we will
focus on monopole-monopole and monopole-dipole interactions, whose parameter
space is displayed in Fig. 1.

Figure 1: QCD axion window for axion-mediated forces vs. current constraints
(full lines) and future experimental sensitivities (dashed lines). Figure
adapted from O’Hare (2020); O’Hare and Vitagliano (2020).

In this figure, the conventional QCD axion region is represented by an orange
band, whose upper (lower) side is set by the scalar coupling in Eq. (3) (Eq.
(4)), while for the pseudo-scalar couplings we employed the predictions of the
DFSZ model at large $\tan\beta$ (cf. discussion below Eq. (2)). Among the
models discussed in this work, we consider three examples, given respectively
by the inclusion of CPV semi-leptonic operators (red), the $Z_{\mathcal{N}}$
axion model with modified $m_{a}$–$f_{a}$ relation (purple), and the PQ-
breaking electron Yukawa scenario (blue). These extend the conventionally
expected range of scalar couplings. The figure should be understood so that
the bands overlap from below, such that e.g. the PQ-breaking model predicts a
coupling between the upper limit of the blue band and down to the lower limit
of the minimal QCD axion band (orange). See previous sections for how the
upper limits on the scalar coupling is obtained in non-minimal QCD axion
models.

The top panel of Fig. 1 displays searches based on monopole-monopole
interactions. In contrast with gravity, these long-range forces deviate from
the inverse-square law and depend on the material’s composition. Tests of the
inverse-square law Chen  _et al._ (2016); Kapner  _et al._ (2007); Lee  _et
al._ (2020); Geraci  _et al._ (2008); Yang  _et al._ (2012); Tan  _et al._
(2020); Ke  _et al._ (2021) can reach parameter space from all three models
presented here. At longer ranges, i.e. lower axion masses, composition
dependence becomes more important than deviations from the inverse-square law.
This enables searches for violations of the equivalence principle Smith  _et
al._ (2000); Bergé  _et al._ (2018) to probe the parameter space offered by
the $Z_{{\cal N}}$ and the PQ-breaking models.

These results suggest that improvements in tests of the equivalence principle
or the inverse square law have the chance to discover these particular types
of QCD axions.

Another interesting opportunity is given by monopole-dipole searches, whose
limits (full lines) and projected sensitivities (dashed lines) are reported in
the middle and lower panels of Fig. 1 for the case of pseudo-scalar axion
couplings to electrons ($g_{ae}^{P}$) and nucleons ($g_{aN}^{P}$),
respectively. In particular, ARIADNE Arvanitaki and Geraci (2014); Geraci  _et
al._ (2018) aims to probe the monopole-dipole force generated by a rotating
source mass and detected via nucleon spins, employing precision magnetometry.
A similar approach is pursued by QUAX-$g_{p}g_{s}$ Crescini  _et al._ (2017a,
b, 2022), using instead electron spins. Several other experiments have
established bounds or are attempting to improve the sensitivity to either
$g_{aX}^{S}g_{ae}^{P}$ Crescini  _et al._ (2022); Hoedl  _et al._ (2011);
Terrano  _et al._ (2015); Heckel  _et al._ (2008); Wineland  _et al._ (1991);
Lee  _et al._ (2018); Fan and Reig (2023); Agrawal  _et al._ (2023a) or
$g_{aX}^{S}g_{aN}^{P}$ Wei  _et al._ (2023); Agrawal  _et al._ (2023b)
couplings. Regardless of which pseudo-scalar coupling is employed, our
scenarios considerably improve the prospects for monopole-dipole searches. The
projected reaches of QUAX-$g_{p}g_{s}$ as well as proposals based on ultracold
molecules Agrawal  _et al._ (2023a) and proton storage rings Agrawal  _et al._
(2023b) can probe parameter space beyond the traditional QCD band which is
populated by the models studied here, while ARIADNE would be able to test wide
ranges of parameter space, also inside the minimal QCD axion band.

Conclusions. In this paper, we have reconsidered the UV origin of the
$g^{S}_{af}$ coupling and identified various mechanisms based on new sources
of CP or PQ breaking. These mechanisms can significantly relax EDM
constraints, leading to a substantial redefinition of the traditional QCD
axion window for axion-mediated forces, as shown in Fig. 1. Our findings
suggest an expanded QCD axion parameter space, providing strong motivation for
pushing the current limits on the search for new macroscopic forces.

Acknowledgments. We thank Antonio Rodríguez-Sánchez for useful discussions.
The work of LDL, HG and PS is supported by the European Union – NextGeneration
EU and by the University of Padua under the 2021 STARS Grants@Unipd programme
(Acronym and title of the project: CPV-Axion – Discovering the CP-violating
axion) as well as by the European Union – Next Generation EU and by the
Italian Ministry of University and Research (MUR) via the PRIN 2022 project n.
2022K4B58X – AxionOrigins.

## References

  * Peccei and Quinn (1977a) R. D. Peccei and H. R. Quinn, Phys. Rev. Lett. 38, 1440 (1977a).
  * Peccei and Quinn (1977b) R. D. Peccei and H. R. Quinn, Phys. Rev. D 16, 1791 (1977b).
  * Weinberg (1978) S. Weinberg, Phys. Rev. Lett. 40, 223 (1978).
  * Wilczek (1978) F. Wilczek, Phys. Rev. Lett. 40, 279 (1978).
  * Abel  _et al._ (2020) C. Abel _et al._ , Phys. Rev. Lett. 124, 081803 (2020), arXiv:2001.11966 [hep-ex] .
  * Vafa and Witten (1984) C. Vafa and E. Witten, Phys. Rev. Lett. 53, 535 (1984).
  * Georgi and Randall (1986) H. Georgi and L. Randall, Nucl. Phys. B276, 241 (1986).
  * Ellis and Gaillard (1979) J. R. Ellis and M. K. Gaillard, Nucl. Phys. B150, 141 (1979).
  * Khriplovich (1986) I. B. Khriplovich, Phys. Lett. B 173, 193 (1986).
  * Gérard and Mertens (2012) J.-M. Gérard and P. Mertens, Phys. Lett. B 716, 316 (2012), arXiv:1206.0914 [hep-ph] .
  * Kallosh  _et al._ (1995) R. Kallosh, A. D. Linde, D. A. Linde, and L. Susskind, Phys. Rev. D 52, 912 (1995), arXiv:hep-th/9502069 .
  * Moody and Wilczek (1984) J. E. Moody and F. Wilczek, Phys. Rev. D30, 130 (1984).
  * O’Hare and Vitagliano (2020) C. A. J. O’Hare and E. Vitagliano, Phys. Rev. D 102, 115026 (2020), arXiv:2010.03889 [hep-ph] .
  * Raffelt (2012) G. Raffelt, Phys. Rev. D 86, 015001 (2012), arXiv:1205.1776 [hep-ph] .
  * Irastorza and Redondo (2018) I. G. Irastorza and J. Redondo, Prog. Part. Nucl. Phys. 102, 89 (2018), arXiv:1801.08127 [hep-ph] .
  * Sikivie (2021) P. Sikivie, Rev. Mod. Phys. 93, 015004 (2021), arXiv:2003.02206 [hep-ph] .
  * Pospelov (1998) M. Pospelov, Phys. Rev. D 58, 097703 (1998), arXiv:hep-ph/9707431 .
  * Bertolini  _et al._ (2021) S. Bertolini, L. Di Luzio, and F. Nesti, Phys. Rev. Lett. 126, 081801 (2021), arXiv:2006.12508 [hep-ph] .
  * Okawa  _et al._ (2022) S. Okawa, M. Pospelov, and A. Ritz, Phys. Rev. D 105, 075003 (2022), arXiv:2111.08040 [hep-ph] .
  * Dekens  _et al._ (2022) W. Dekens, J. de Vries, and S. Shain, JHEP 07, 014 (2022), arXiv:2203.11230 [hep-ph] .
  * Plakkot  _et al._ (2023) V. Plakkot, W. Dekens, J. de Vries, and S. Shain, JHEP 11, 012 (2023), arXiv:2306.07065 [hep-ph] .
  * Di Luzio  _et al._ (2023) L. Di Luzio, H. Gisbert, G. Levati, P. Paradisi, and P. Sørensen, (2023), arXiv:2312.17310 [hep-ph] .
  * Zhitnitsky (1980) A. R. Zhitnitsky, Sov. J. Nucl. Phys. 31, 260 (1980).
  * Dine  _et al._ (1981) M. Dine, W. Fischler, and M. Srednicki, Phys. Lett. B 104, 199 (1981).
  * Di Luzio  _et al._ (2020) L. Di Luzio, M. Giannotti, E. Nardi, and L. Visinelli, Phys. Rept. 870, 1 (2020), arXiv:2003.01100 [hep-ph] .
  * O’Hare (2020) C. O’Hare, “cajohare/axionlimits: Axionlimits,” https://cajohare.github.io/AxionLimits/ (2020).
  * Pich and de Rafael (1991) A. Pich and E. de Rafael, Nucl. Phys. B 367, 313 (1991).
  * Arvanitaki and Geraci (2014) A. Arvanitaki and A. A. Geraci, Phys. Rev. Lett. 113, 161801 (2014), arXiv:1403.1290 [hep-ph] .
  * Geraci  _et al._ (2018) A. A. Geraci _et al._ (ARIADNE), Springer Proc. Phys. 211, 151 (2018), arXiv:1710.05413 [astro-ph.IM] .
  * Cirigliano  _et al._ (2017) V. Cirigliano, W. Dekens, J. de Vries, and E. Mereghetti, Phys. Lett. B 767, 1 (2017), arXiv:1612.03914 [hep-ph] .
  * Bertolini  _et al._ (2020) S. Bertolini, A. Maiezza, and F. Nesti, Phys. Rev. D 101, 035036 (2020), arXiv:1911.09472 [hep-ph] .
  * Graner  _et al._ (2016) B. Graner, Y. Chen, E. G. Lindahl, and B. R. Heckel, Phys. Rev. Lett. 116, 161601 (2016), [Erratum: Phys.Rev.Lett. 119, 119901 (2017)], arXiv:1601.04339 [physics.atom-ph] .
  * Flambaum  _et al._ (2020) V. V. Flambaum, M. Pospelov, A. Ritz, and Y. V. Stadnik, Phys. Rev. D 102, 035001 (2020), arXiv:1912.13129 [hep-ph] .
  * Andreev  _et al._ (2018) V. Andreev _et al._ (ACME), Nature 562, 355 (2018).
  * Bishof  _et al._ (2016) M. Bishof _et al._ , Phys. Rev. C 94, 025501 (2016), arXiv:1606.04931 [nucl-ex] .
  * Hudson  _et al._ (2011) J. J. Hudson, D. M. Kara, I. J. Smallman, B. E. Sauer, M. R. Tarbutt, and E. A. Hinds, Nature 473, 493 (2011).
  * Cairncross  _et al._ (2017) W. B. Cairncross, D. N. Gresh, M. Grau, K. C. Cossel, T. S. Roussy, Y. Ni, Y. Zhou, J. Ye, and E. A. Cornell, Phys. Rev. Lett. 119, 153001 (2017), arXiv:1704.07928 [physics.atom-ph] .
  * Allwicher  _et al._ (2023) L. Allwicher, D. A. Faroughy, F. Jaffredo, O. Sumensari, and F. Wilsch, JHEP 03, 064 (2023), arXiv:2207.10714 [hep-ph] .
  * Zhang (2023) Y. Zhang, Phys. Rev. D 107, 055025 (2023), arXiv:2209.09429 [hep-ph] .
  * Zhang (2024) Y. Zhang, Phys. Rev. Lett. 132, 081003 (2024), arXiv:2305.15495 [hep-ph] .
  * Sikivie (1982) P. Sikivie, Phys. Rev. Lett. 48, 1156 (1982).
  * Di Luzio  _et al._ (2021a) L. Di Luzio, R. Gröber, and P. Paradisi, Phys. Rev. D 104, 095027 (2021a), arXiv:2010.13760 [hep-ph] .
  * Hook (2018) A. Hook, Phys. Rev. Lett. 120, 261802 (2018), arXiv:1802.10093 [hep-ph] .
  * Di Luzio  _et al._ (2021b) L. Di Luzio, B. Gavela, P. Quilez, and A. Ringwald, JHEP 05, 184 (2021b), arXiv:2102.00012 [hep-ph] .
  * Di Luzio  _et al._ (2021c) L. Di Luzio, B. Gavela, P. Quilez, and A. Ringwald, JCAP 10, 001 (2021c), arXiv:2102.01082 [hep-ph] .
  * Hook and Huang (2018) A. Hook and J. Huang, JHEP 06, 036 (2018), arXiv:1708.08464 [hep-ph] .
  * Balkin  _et al._ (2024) R. Balkin, J. Serra, K. Springmann, S. Stelzl, and A. Weiler, Phys. Rev. D 109, 095032 (2024), arXiv:2211.02661 [hep-ph] .
  * Chen  _et al._ (2016) Y. J. Chen, W. K. Tham, D. E. Krause, D. Lopez, E. Fischbach, and R. S. Decca, Phys. Rev. Lett. 116, 221102 (2016), arXiv:1410.7267 [hep-ex] .
  * Kapner  _et al._ (2007) D. J. Kapner, T. S. Cook, E. G. Adelberger, J. H. Gundlach, B. R. Heckel, C. D. Hoyle, and H. E. Swanson, Phys. Rev. Lett. 98, 021101 (2007), arXiv:hep-ph/0611184 .
  * Lee  _et al._ (2020) J. G. Lee, E. G. Adelberger, T. S. Cook, S. M. Fleischer, and B. R. Heckel, Phys. Rev. Lett. 124, 101101 (2020), arXiv:2002.11761 [hep-ex] .
  * Geraci  _et al._ (2008) A. A. Geraci, S. J. Smullin, D. M. Weld, J. Chiaverini, and A. Kapitulnik, Phys. Rev. D 78, 022002 (2008), arXiv:0802.2350 [hep-ex] .
  * Yang  _et al._ (2012) S.-Q. Yang, B.-F. Zhan, Q.-L. Wang, C.-G. Shao, L.-C. Tu, W.-H. Tan, and J. Luo, Phys. Rev. Lett. 108, 081101 (2012).
  * Tan  _et al._ (2020) W.-H. Tan _et al._ , Phys. Rev. Lett. 124, 051301 (2020).
  * Ke  _et al._ (2021) J. Ke, J. Luo, C.-G. Shao, Y.-J. Tan, W.-H. Tan, and S.-Q. Yang, Phys. Rev. Lett. 126, 211101 (2021).
  * Smith  _et al._ (2000) G. L. Smith, C. D. Hoyle, J. H. Gundlach, E. G. Adelberger, B. R. Heckel, and H. E. Swanson, Phys. Rev. D 61, 022001 (2000), arXiv:2405.10982 [gr-qc] .
  * Bergé  _et al._ (2018) J. Bergé, P. Brax, G. Métris, M. Pernot-Borràs, P. Touboul, and J.-P. Uzan, Phys. Rev. Lett. 120, 141101 (2018), arXiv:1712.00483 [gr-qc] .
  * Crescini  _et al._ (2017a) N. Crescini, C. Braggio, G. Carugno, P. Falferi, A. Ortolan, and G. Ruoso, Nucl. Instrum. Meth. A 842, 109 (2017a), arXiv:1606.04751 [physics.ins-det] .
  * Crescini  _et al._ (2017b) N. Crescini, C. Braggio, G. Carugno, P. Falferi, A. Ortolan, and G. Ruoso, Phys. Lett. B 773, 677 (2017b), arXiv:1705.06044 [hep-ex] .
  * Crescini  _et al._ (2022) N. Crescini, G. Carugno, P. Falferi, A. Ortolan, G. Ruoso, and C. C. Speake, Phys. Rev. D 105, 022007 (2022), arXiv:2011.07100 [hep-ex] .
  * Hoedl  _et al._ (2011) S. A. Hoedl, F. Fleischer, E. G. Adelberger, and B. R. Heckel, Phys. Rev. Lett. 106, 041801 (2011).
  * Terrano  _et al._ (2015) W. A. Terrano, E. G. Adelberger, J. G. Lee, and B. R. Heckel, Phys. Rev. Lett. 115, 201801 (2015), arXiv:1508.02463 [hep-ex] .
  * Heckel  _et al._ (2008) B. R. Heckel, E. G. Adelberger, C. E. Cramer, T. S. Cook, S. Schlamminger, and U. Schmidt, Phys. Rev. D 78, 092006 (2008), arXiv:0808.2673 [hep-ex] .
  * Wineland  _et al._ (1991) D. J. Wineland, J. J. Bollinger, D. J. Heinzen, W. M. Itano, and M. G. Raizen, Phys. Rev. Lett. 67, 1735 (1991).
  * Lee  _et al._ (2018) J. Lee, A. Almasi, and M. Romalis, Phys. Rev. Lett. 120, 161801 (2018), arXiv:1801.02757 [hep-ex] .
  * Fan and Reig (2023) X. Fan and M. Reig, (2023), arXiv:2310.18797 [hep-ph] .
  * Agrawal  _et al._ (2023a) P. Agrawal, N. R. Hutzler, D. E. Kaplan, S. Rajendran, and M. Reig, (2023a), arXiv:2309.10023 [hep-ph] .
  * Wei  _et al._ (2023) K. Wei, T. Zhao, X. Fang, Z. Xu, C. Liu, Q. Cao, A. Wickenbrock, Y. Hu, W. Ji, and D. Budker, Phys. Rev. Lett. 130, 063201 (2023), arXiv:2210.09027 [physics.atom-ph] .
  * Agrawal  _et al._ (2023b) P. Agrawal, D. E. Kaplan, O. Kim, S. Rajendran, and M. Reig, Phys. Rev. D 108, 015017 (2023b), arXiv:2210.17547 [hep-ph] .
  * Scherer and Schindler (2012) S. Scherer and M. R. Schindler,  _A Primer for Chiral Perturbation Theory_, Vol. 830 (Springer Berlin, Heidelberg, 2012).
  * Hoferichter  _et al._ (2015) M. Hoferichter, J. Ruiz de Elvira, B. Kubis, and U.-G. Meißner, Phys. Rev. Lett. 115, 092301 (2015), arXiv:1506.04142 [hep-ph] .
  * Hoferichter  _et al._ (2016) M. Hoferichter, J. Ruiz de Elvira, B. Kubis, and U.-G. Meißner, Phys. Lett. B 760, 74 (2016), arXiv:1602.07688 [hep-lat] .
  * An  _et al._ (2010) H. An, X. Ji, and F. Xu, JHEP 02, 043 (2010), arXiv:0908.2420 [hep-ph] .
  * Guo and Meissner (2012) F.-K. Guo and U.-G. Meissner, JHEP 12, 097 (2012), arXiv:1210.5887 [hep-ph] .

##  Appendix A Baryon-meson couplings contributions to EDMs

In this Appendix, we discuss the contributions of baryon-meson couplings to
the neutron and proton EDMs. The leading order (LO) baryon chiral Lagrangian
can be written as

$\displaystyle\mathcal{L}_{\rm B}$ $\displaystyle={\rm
Tr}\left[\overline{B}i\gamma^{\mu}(\partial_{\mu}B+[\Gamma_{\mu},\,B])-M_{B}\overline{B}B\right]-\frac{D}{2}{\rm
tr}\left[\overline{B}\gamma^{\mu}\gamma_{5}\\{\xi_{\mu},\,B\\}\right]-\frac{F}{2}{\rm
tr}\left[\overline{B}\gamma^{\mu}\gamma_{5}[\xi_{\mu},\,B]\right]$
$\displaystyle-\frac{\lambda}{2}\ {\rm tr}\left[\xi_{\mu}\right]{\rm
tr}\left[\overline{B}\gamma^{\mu}\gamma_{5}B\right]+b_{D}\ {\rm
tr}\left[\overline{B}\\{\chi_{+},\,B\\}\right]+b_{F}\ {\rm
tr}\left[\overline{B}[\chi_{+},\,B]\right]+b_{0}\ {\rm
tr}\left[\chi_{+}\right]{\rm tr}\left[\overline{B}B\right]\,,$ (22)

where we employed the definitions

$\displaystyle
B=\begin{pmatrix}\frac{1}{\sqrt{2}}\Sigma^{0}+\frac{1}{\sqrt{6}}\Lambda^{0}&\Sigma^{+}&p\\\
\Sigma^{-}&-\frac{1}{\sqrt{2}}\Sigma^{0}+\frac{1}{\sqrt{6}}\Lambda^{0}&n\\\
\Xi^{-}&\Xi^{0}&-\frac{2}{\sqrt{6}}\Lambda^{0}\\\ \end{pmatrix},\hskip
20.00003ptU=\xi_{R}\,\xi_{L}^{\dagger}\qquad(\xi_{R}=\xi_{L}^{\dagger})~{},$
(23)

and

$\displaystyle\Gamma_{\mu}\equiv{}$
$\displaystyle\frac{1}{2}\xi_{R}^{\dagger}(\partial_{\mu}-i\,r_{\mu})\xi_{R}+\frac{1}{2}\xi_{L}^{\dagger}(\partial_{\mu}-i\,l_{\mu})\xi_{L}\,,$
(24) $\displaystyle\xi_{\mu}\equiv{}$ $\displaystyle
i\xi_{R}^{\dagger}(\partial_{\mu}-i\,r_{\mu})\xi_{R}-i\xi_{L}^{\dagger}(\partial_{\mu}-i\,l_{\mu})\xi_{L}\,,$
(25) $\displaystyle\chi_{+}\equiv{}$
$\displaystyle\xi_{L}^{\dagger}\chi\xi_{R}+\xi_{R}^{\dagger}\chi^{\dagger}\xi_{L}\,.$
(26)

Here, $M_{B}$ denotes the baryon mass in the chiral limit, while $\chi=2B_{0}\
{\rm diag}\\{m_{u},m_{d},m_{s}\\}$ is the chiral spurion including the quark
masses. In Eq. (22), the interaction terms proportional to $D$, $F$ and
$\lambda$ are CP conserving, while those proportional to $b_{D}$, $b_{F}$ and
$b_{0}$ violate CP. The $D$ and $F$ LECs are extracted from baryon semi-
leptonic decays, and they read at tree level $D\simeq 0.8$ and $F\simeq 0.5$
Scherer and Schindler (2012). The LEC $B_{0}$ is given by
$B_{0}=m_{\pi}^{2}/(m_{d}+m_{u})$, while $b_{D,F}$ are determined from the
baryon octet mass splittings, $b_{D}\simeq 0.07\,\rm GeV^{-1}$,
$b_{F}\simeq-0.21\,\rm GeV^{-1}$ at LO Pich and de Rafael (1991). The value of
$b_{0}$ is determined from the pion-nucleon sigma-term as
$b_{0}\simeq-\sigma_{\pi N}/4m_{\pi}^{2}$. From Hoferichter  _et al._ (2015,
2016) one obtains $b_{0}=-0.76\pm 0.04\,\rm GeV^{-1}$ at 90% confidence level.
The LEC $\lambda$ is unknown and is set to zero. In the main text, we also
defined $b_{\pm}=b_{D}\pm b_{F}$.

The QCD $\theta$-term plus the axion is rotated away by an appropriate axion-
dependent chiral rotation of the quark fields, $q_{L}\to
e^{-i\,\alpha_{q}}q_{L}$, $q_{R}\to e^{i\,\alpha_{q}}q_{R}$, with
$\alpha_{q}=\left(\theta+\frac{a}{f_{a}}\right)m_{*}/2m_{q}$ Pich and de
Rafael (1991). By applying this diagonal $U(3)_{A}$ field transformation on
the $\xi_{L,R}$ and the baryon chiral fields, the axion is effectively
included in the meson Lagrangian as well as in the baryon Lagrangian (22).
With this choice of $\alpha_{q}$ the axion does not mix with $\pi^{0}$ and
$\eta_{8}$.

In the presence of new physics in the $\chi$PT Lagrangian, one has to further
rotate the $U$ fields to have $\left\langle U\right\rangle=1$, as well as to
shift the axion field. This again amounts to a diagonal rotation (we consider
strangeness preserving interactions here) so that one can consider altogether
three generic $\alpha_{q}$, incorporating the mesons and axion VEVs, as in Eq.
(6) in the main text. As already mentioned, this _global_ chiral rotation is
effectively equivalent to a rotation of the spurions of chiral
$SU(3)_{V}\times U(3)_{A}$, namely a rotation of the quark masses in the meson
and baryon chiral Lagrangian (22), as follows: $m_{q}\to
m_{q}e^{2i\alpha_{q}}$.

It is this phase rotation that generates, from the baryon Lagrangian, the
various CPV baryon interactions with the physical meson fields Pich and de
Rafael (1991); An  _et al._ (2010). We parametrize their couplings as follows:

$\displaystyle\mathcal{L}_{\text{B}\chi\text{PT}}^{\rm\,CPV}\,\supset\,\overline{g}_{ABC}\,A\,B\,C~{},$
(27)

where $A$, $B$ are baryon fields and $C$ is a meson, and their expressions are
reported below.

The neutron and proton EDMs are generated at loop level and can be expressed,
following e.g. Ref. Guo and Meissner (2012), in terms of these couplings and
loop functions, which are expanded at leading order in the meson masses. We
write $d_{n,p}\,=\,\sum_{i}\,d_{n,p}^{(i)}$, where the index $i$ runs over all
baryon-meson couplings contributing to the neutron and proton EDMs.
Specifically, for the neutron EDM, only two couplings contribute Guo and
Meissner (2012):

$\displaystyle d_{n}^{(\overline{n}p\pi^{-})}\,$
$\displaystyle=\,\frac{e\,(F\,+\,D)\,\overline{g}_{\overline{n}p\pi^{-}}\,(\pi\,m_{\pi}\,-\,2\,m_{N}\,\log(m_{\pi}^{2}/m_{N}^{2}))}{16\,\sqrt{2}\,F_{\pi}\,m_{N}\,\pi^{2}}~{},$
(28) $\displaystyle d_{n}^{(\overline{n}\Sigma^{-}K^{+})}\,$
$\displaystyle=\,\frac{e\,(F\,-\,D)\,\overline{g}_{\overline{n}\Sigma^{-}K^{+}}\,(\pi\,m_{K}\,-\,2\,m_{N}\,\log(m_{K}^{2}/m_{N}^{2}))}{16\,\sqrt{2}\,F_{\pi}\,m_{N}\,\pi^{2}}~{},$
(29)

while more couplings contribute to the proton EDM due to its electric charge:

$\displaystyle d_{p}^{(\overline{p}p\pi^{0})}\,$
$\displaystyle=\,-\,\frac{e\,(D\,+\,F)\,\overline{g}_{\overline{p}p\pi^{0}}\,m_{\pi}}{16\,\sqrt{2}\,F_{\pi}\,m_{N}\,\pi}~{},$
(30) $\displaystyle d_{p}^{(n\overline{p}\pi^{+})}\,$
$\displaystyle=\,-\,\frac{e\,(D\,+\,F)\,\overline{g}_{n\overline{p}\pi^{+}}\,(3\pi\,m_{\pi}\,-\,2\,m_{N}\,\log(m_{\pi}^{2}/m_{N}^{2}))}{16\,\sqrt{2}\,F_{\pi}\,m_{N}\,\pi^{2}}~{},$
(31) $\displaystyle d_{p}^{(\Sigma^{+}\overline{p}K^{0})}\,$
$\displaystyle=-\frac{e\,(D\,-\,F)\,\overline{g}_{\Sigma^{+}\overline{p}K^{0}}\,m_{K}}{8\,\sqrt{2}\,F_{\pi}\,m_{N}\,\pi}~{},$
(32) $\displaystyle d_{p}^{(\overline{p}\Lambda^{0}K^{+})}\,$
$\displaystyle=-\,\frac{e\,(D\,+\,3\,F)\,\overline{g}_{\overline{p}\Lambda^{0}K^{+}}\,(3\pi\,m_{K}\,-\,2\,m_{N}\,\log(m_{K}^{2}/m_{N}^{2}))}{96\,\sqrt{2}\,F_{\pi}\,m_{N}\,\pi^{2}}~{},$
(33) $\displaystyle d_{p}^{(\Sigma^{0}\overline{p}K^{+})}\,$
$\displaystyle=-\,\frac{e\,(D\,-\,F)\,\overline{g}_{\Sigma^{0}\overline{p}K^{+}}\,(3\pi\,m_{K}\,-\,2\,m_{N}\,\log(m_{K}^{2}/m_{N}^{2}))}{32\,\sqrt{2}\,F_{\pi}\,m_{N}\,\pi^{2}}~{},$
(34) $\displaystyle d_{p}^{(\overline{p}p\eta_{8})}\,$
$\displaystyle=\,-\,\frac{e\,(D\,-\,3\,F)\,\overline{g}_{\overline{p}p\eta_{8}}\,m_{\eta_{8}}}{48\,\sqrt{2}\,F_{\pi}\,m_{N}\,\pi}~{},$
(35) $\displaystyle d_{p}^{(\overline{p}p\eta_{0})}\,$
$\displaystyle=\,-\,\frac{e\,(2\,D\,+\,3\,\lambda)\,\overline{g}_{\overline{p}p\eta_{0}}\,m_{\eta_{0}}}{24\,\sqrt{2}\,F_{\pi}\,m_{N}\,\pi}~{}.$
(36)

For the couplings in terms of phases, we find:

$\displaystyle\overline{g}_{\overline{n}p\pi^{-}}\,$
$\displaystyle=\,-\,\frac{4\,\sqrt{2}\,B_{0}\,(b_{D}\,+\,b_{F})\,(m_{d}\,\alpha_{d}\,+\,m_{u}\,\alpha_{u})}{F_{\pi}}~{},$
(37) $\displaystyle\overline{g}_{\overline{n}\Sigma^{-}K^{+}}\,$
$\displaystyle=\,-\,\frac{4\,\sqrt{2}\,B_{0}\,(b_{D}\,-\,b_{F})\,(m_{s}\,\alpha_{s}\,+\,m_{u}\,\alpha_{u})}{F_{\pi}}~{}.$
(38)

and

$\displaystyle\overline{g}_{\overline{p}p\pi^{0}}\,$
$\displaystyle=\,\frac{8B_{0}[b_{0}\,(\,m_{d}\,\alpha_{d}\,-\,m_{u}\,\alpha_{u})-(b_{D}\,+\,b_{F})\,m_{u}\,\alpha_{u}]}{F_{\pi}}~{},$
(39) $\displaystyle\overline{g}_{nn\pi^{0}}\,$
$\displaystyle=\,\frac{8B_{0}[b_{0}\,(m_{d}\,\alpha_{d}\,-\,m_{u}\,\alpha_{u})+(b_{D}\,+\,b_{F})\,m_{d}\,\alpha_{d}]}{F_{\pi}}~{},$
(40) $\displaystyle\overline{g}_{n\overline{p}\pi^{+}}\,$
$\displaystyle=\,\,-\,\frac{4\,\sqrt{2}\,B_{0}\,(b_{D}\,+\,b_{F})\,(m_{d}\,\alpha_{d}\,+\,m_{u}\,\alpha_{u})}{F_{\pi}}~{},$
(41) $\displaystyle\overline{g}_{\Sigma^{+}\overline{p}K^{0}}\,$
$\displaystyle=\,\,-\,\frac{4\,\sqrt{2}\,B_{0}\,(b_{D}\,-\,b_{F})\,(m_{d}\,\alpha_{d}\,+\,m_{s}\,\alpha_{s})}{F_{\pi}}~{},$
(42) $\displaystyle\overline{g}_{\overline{p}\Lambda^{0}K^{+}}\,$
$\displaystyle=\,\,-\,\frac{4\,B_{0}\,(b_{D}\,+\,3\,b_{F})\,(m_{s}\,\alpha_{s}\,+\,m_{u}\,\alpha_{u})}{\sqrt{3}\,F_{\pi}}~{},$
(43) $\displaystyle\overline{g}_{\Sigma^{0}\overline{p}K^{+}}\,$
$\displaystyle=\,\,-\,\frac{4\,\sqrt{2}\,B_{0}\,(b_{D}\,-\,b_{F})\,(m_{s}\,\alpha_{s}\,+\,m_{u}\,\alpha_{u})}{F_{\pi}}~{},$
(44) $\displaystyle\overline{g}_{\overline{p}p\eta_{8}}\,$
$\displaystyle=\,\frac{8\,B_{0}\,\big{[}b_{0}(2\,m_{s}\,\alpha_{s}-m_{u}\alpha_{u}-m_{d}\alpha_{d})+\,2(\,b_{D}-b_{F})\,m_{s}\,\alpha_{s}\,-\,(b_{D}\,+\,b_{F})\,m_{u}\,\alpha_{u}\big{]}}{\sqrt{3}\,F_{\pi}}~{},$
(45) $\displaystyle\overline{g}_{\overline{p}p\eta_{0}}\,$
$\displaystyle=\,-\,\frac{8\,\sqrt{\frac{2}{3}}\,B_{0}\,\big{[}b_{0}(\,m_{u}\,\alpha_{u}+m_{d}\,\alpha_{d}\,+m_{s}\,\alpha_{s})+(b_{D}-\,b_{F})\,m_{s}\,\alpha_{s}\,+\,(b_{D}\,+\,b_{F})\,m_{u}\,\alpha_{u}\big{]}}{\sqrt{3}\,F_{\pi}}\,.$
(46)

In the above formulae, we have taken the large $N_{C}$-limit $F_{\pi}\simeq
F_{0}$, with $F_{\pi}$ and $F_{0}$ being the pion and the eta decay constants,
respectively.

Taking these results into account, we obtain the relevant EDMs in terms of
generic phases:

$\displaystyle d_{p}\,$
$\displaystyle=\,\,\left\\{\alpha_{u}\,\left(-1.72^{+0.19}_{-0.24}\right)\times
10^{-15}\,+\,\alpha_{d}\,\left(-2.63^{+0.14}_{-0.16}\right)\times
10^{-15}\,+\,\alpha_{s}\,\left(-1.31^{+0.25}_{-0.30}\right)\times
10^{-14}\right\\}\,e\,\text{cm}\,,$ (47) $\displaystyle d_{n}\,$
$\displaystyle=\,\left\\{\alpha_{u}\,\left(6.7^{+0.7}_{-0.9}\right)\times
10^{-16}\,+\,\alpha_{d}\,\left(10.6^{+0.6}_{-0.7}\right)\times
10^{-16}\,+\,\alpha_{s}\,\left(7.1^{+0.5}_{-0.5}\right)\times
10^{-15}\right\\}\,e\,\text{cm}\,.$ (48)

where hadronic uncertainties from LECs and quark masses have been included. By
considering the impact of the neutron and proton EDMs on the Mercury EDM
Dekens  _et al._ (2022), we find

$\displaystyle
d_{\text{Hg}}\,=\,\left\\{\alpha_{u}\,\left(-2.0^{+0.6}_{-0.7}\right)\times
10^{-19}\,+\,\alpha_{d}\,\left(-3.1^{+1.1}_{-1.2}\right)\times
10^{-19}\,+\,\alpha_{s}\,\left(-23\pm 11\right)\times
10^{-19}\right\\}\,e\,\text{cm}~{}.$ (49)

The neutron and Mercury EDM expressions should be compared with the
experimental bounds, respectively $d_{n}^{<}=1.8\times 10^{-26}\,e\,\text{cm}$
Abel  _et al._ (2020) and $d_{\text{Hg}}^{<}=6.3\times 10^{-30}\,e\,\text{cm}$
Graner  _et al._ (2016).

The ThO EDM bound finally amounts to $\overline{g}_{\pi NN}^{(1)}\lesssim
4\times 10^{-10}$ Flambaum  _et al._ (2020), with

$\overline{g}_{\pi
NN}^{(1)}\equiv\frac{1}{2}(\overline{g}_{nn\pi}+\overline{g}_{pp\pi})=\frac{4\,B_{0}(2\,b_{0}+b_{D}+b_{F})(\alpha_{d}\,m_{d}-\alpha_{u}\,m_{u})}{F_{\pi}}=\left(0.42^{+0.05}_{-0.07}\right)\alpha_{u}+\left(-0.89^{+0.04}_{-0.05}\right)\alpha_{d}\,.$
(50)

Eqs. (48)–(50) translate into the constraints of Eqs. (8)–(10) in the main
text.

We finally look for $\alpha_{u,d,s}$ that, while respecting these constraints,
try to maximize the CPV coupling in Eq. (5), which numerically is

$\displaystyle g_{aX}^{S}$ $\displaystyle=10^{-11}\frac{10^{9}\,{\rm
GeV}}{f_{a}}\big{[}(2.6^{+0.4}_{-0.5})\,\alpha_{u}+(2.6^{+0.4}_{-0.5})\,\alpha_{d}+(1.50^{+0.21}_{-0.27})\,\alpha_{s}\big{]}\,,$
(51)

We find that all $\alpha_{u,d,s}$ are constrained to be of order
$O(10^{-{9\div 10}})$, and lead to a maximal value of

$\left|\frac{g_{aX}^{S}}{g_{aX}^{S,\theta}}\right|\lesssim 10\,,$ (52)

allowing also for the uncertainties in Eqs. (48)–(50).


# Conditional Motion In-betweening

Jihoon Kim<EMAIL_ADDRESS>Taehyun Byun<EMAIL_ADDRESS>Seungyoun Shin<EMAIL_ADDRESS>Jungdam Won<EMAIL_ADDRESS>Sungjoon Choi
<EMAIL_ADDRESS>
###### Abstract

Motion in-betweening (MIB) is a process of generating intermediate skeletal
movement between the given start and target poses while preserving the
naturalness of the motion, such as periodic footstep motion while walking.
Although state-of-the-art MIB methods are capable of producing plausible
motions given sparse key-poses, they often lack the controllability to
generate motions satisfying the semantic contexts required in practical
applications. We focus on the method that can handle pose or semantic
conditioned MIB tasks using a unified model. We also present a motion
augmentation method to improve the quality of pose-conditioned motion
generation via defining a distribution over smooth trajectories. Our proposed
method outperforms the existing state-of-the-art MIB method in pose prediction
errors while providing additional controllability. Our code and results are
available on our project web page: https://jihoonerd.github.io/Conditional-
Motion-In-Betweening

###### keywords:

motion in-betweening , conditional motion generation , generative model ,
motion data augmentation

††journal: Pattern Recognition

[inst1]organization=Department of Artificial Intelligence,addressline=Korea
University, city=Seoul, postcode=02841, country=South Korea

[inst2]organization=Department of Computer Science and
Engineering,addressline=Dongguk University, city=Seoul, postcode=04620,
country=South Korea

[inst3]organization=Meta,addressline=1 Hacker Way, city=Menlo Park,
postcode=94025, state=CA, country=United States

##  1 Introduction

The demand for generating natural and expressive 3D human motion proliferates
in the film and gaming industries. Despite the demand, however, generating
diverse character movements in industries still dominantly relies on Motion
Capture (MoCap) machines and traditional approaches [1, 2] rather than
learning-based approaches due to its inherent complexity. A model that can
synthesize natural human motion with providing controllability can help
professional animators by allowing them to focus on creative and novel motions
by reducing the labor involved in simple and redundant motion creation. One
unique property of human motion is that it is spatio-temporally constrained by
a human’s feasible skeletal (kinematic) structure, which distinguishes itself
from static and structured data such as images. In the context of motion
generation, a wide range of studies [3, 4] has focused on forecasting natural
motion-frames (i.e., motion prediction) from a given key-pose or multiple sets
of key-poses. Others [5, 6, 7] investigate methods to generate motion from
given semantics such as textual action description. In this paper, we focus on
a motion in-betweening (MIB) problem, where MIB is a process of generating
skeletal motions that naturally interpolates a given set of key-poses,
reducing the great amount of time and manual effort required for animators.

(a) Ground truth

(b) Motion in-betweening

(c) Pose-conditioned motion in-betweening

(d) Semantic-conditioned motion in-betweening

Figure 1: (a) Ground Truth motion is visualized from human Motion Capture
(MoCap) data. (b) Motion in-betweening is a task that connects given poses
with plausible motion. (c) Pose-conditioned motion in-betweening connects
given starting and target poses while satisfying the given interim pose. (d)
Semantic-conditioned motion in-betweening connects given starting and target
poses while satisfying given semantic information, such as walking, jumping,
and dancing.

Recent studies [8, 9] have shown that learning-based methods can provide
plausible solutions for MIB given sparse key-poses. However, they often lack
controllability in human motion generation. This paper presents a conditional
motion in-betweening (CMIB) method that can handle two types of conditioned
motion generation tasks: pose-conditioned and semantic/conditioned MIB. First,
the pose-conditioned MIB generates a sequence of poses from the start, anchor,
and target poses so that the generated motion naturally interpolates the given
poses while preserving the naturalness of the motion. We also present a motion
augmentation method to further provide the pose-controllability beyond the
given motion dataset. In particular, we augment the motion trajectories by
sampling the root trajectories from a Gaussian random path distribution [10]
which defines a probability distribution over smooth trajectories in
reproducing kernel Hilbert space. Finally, semantic-conditioned MIB enables
users to choose desired semantics at the inference stage. These conditioning
problems are described in Figure 1.

We propose a CMIB method that integrates pose and semantic CMIB in a single
Transformer encoder-based architecture. We interpret MIB as a masked motion
modeling problem and introduce a randomized shuffled anchor pose which makes
our motion encoder perform pose-conditioned MIB. We also introduce a semantic
embedding token by prepending a sequence’s semantic context to motion
representation. To the best of our knowledge, we are the first to propose a
controllable motion generation method in the MIB framework.

Our main contributions are summarized as follows:

  * 1.

We introduce a Transformer encoder-based CMIB model that can perform the
following conditional motion generation tasks using a single model:

    * (a)

Pose-conditioned: Generate a motion sequence that satisfies the given anchor
pose while interpolating start and target poses

    * (b)

Semantic-conditioned: Generate a motion sequence with given semantic
information (e.g., walk, run, or dance)

  * 2.

We propose a motion data augmentation strategy that can generate various
trajectories from existing motion for pose-conditioned MIB tasks.

  * 3.

Our proposed method outperforms the existing state-of-the-art MIB method with
additional capabilities of conditioned motion generation. Furthermore, we also
present a performance evaluation measure to validate semantic-conditioned MIB
tasks.

##  2 Related Work

Motion Prediction Motion prediction refers to forecasting plausible motion
from previous pose(s). The recent development of deep learning-based
approaches has allowed significant advancement in motion synthesis. Recent
studies adopted Recurrent Neural Networks (RNN) to address motion prediction
problems. Fragkiadaki et al. [3] introduced a three-layer long short-term
memory (LSTM-3LR) network and encoder-recurrent-decoder (ERD) structure, which
takes advantage of the autoregressive approach in latent dimension formulated
from nonlinear encoder and decoder. Martinez et al. [11] suggested a sequence
to sequence (seq2seq) architecture with residual connections in decoders.
Ghosh et al. [4] integrated dropout autoencoder (DAE) to LSTM-3LR to mitigate
the accumulation of error in RNN. Gui et al. [12] introduced a new loss based
on geodesic loss instead of conventional Euclidean loss with adversarial
training to predict human motion. Harvey et al. [13] improved ERD by employing
additional encoder blocks to inject various contexts to RNN. Aksan et al. [14]
decomposed motion prediction into joint-level prediction by modeling
dependencies across joints with a structured prediction layer. Rossi et al.
[15] have studied motion prediction from the perspective of trajectory
prediction by leveraging both LSTM and generative adversarial network (GAN).

However, even with the help of the improved RNN structure, long-term
prediction is an issue due to the inherent error accumulation problem of RNN,
also with difficulty in parallelization. Aksan et al. [16] introduced self-
attention to directly attend to the previous context and capture dependencies
across poses and presented that the Transformer architecture can produce a
long motion sequence with reduced error accumulation. Martinez et al. [17]
proposed a seq2seq Transformer encoder and decoder model with Graph
Convolutional Networks (GCN) before and after the Transformer block.

The conditioned motion generation focuses on generating motion from given
semantic information, such as music or action. Semantic information of human
motion has been widely studied in action recognition tasks, but conditioned
motion generation has a difference in that it should be able to generate
desired motions for given semantics. Won et al. [5] designed a generate-and-
rank approach to choreograph human motion from high-level semantics. Ahn et
al. [6] proposed a method to generate motion from text input. They trained an
autoencoder between the language and action, and the language encoder is
separately trained again in an adversarial manner with the attention
mechanism. Henter et al. [18] set the stage for using normalizing flows on
probabilistic and controllable motion generation. Ling et al. [19] proposed
motion variational autoencoder (motion-VAE), which is based on conditional
variational autoencoder (CVAE), to achieve controllable motion generation. Guo
et al. [7] also studied action-conditioned motion generation, allowing the
RNN-based architecture to learn action by providing action code with pose
vector while training CVAE. Since MIB is also a part of motion generation,
studies on motion prediction form an important foundation for our research.

Motion in-betweening In this work, we consider motion in-betweening (MIB) as a
motion generation task, completing natural motion from sparse pose
information. This task is analogous to image in-painting [20]. However, MIB
has an additional difficulty in that it has to synthesize spatio-temporal
motion data. More closely, the video in-painting [21, 22] shares similarities
in that it should take both spatial and temporal aspects into consideration to
connect sparse information naturally.

Frame-based video interpolation [23] also addresses interpolation problems,
connecting sparse image frames with image sequences learned from videos.
Image-level interpolation has the advantage that it can handle not only motion
but also spatially adjacent information such as background or interacting
objects. However, utilizing human motion for 3D applications is difficult with
this approach since motion is entangled with other information in image space
and lacks depth information, which is essential for 3D motion representation.

Rose et al. [24] employed radial basis function (RBF) to interpolate
parameterized motions. Mukai et al. [25] proposed a statistical prediction
method which optimizes frame-level interpolating kernels for a given
parametric space to perform MIB. Lehrmann et al. [26] have shown the
possibility of Markov models in MIB tasks.

MIB can also be viewed as a boundary value problem (BVP) for multi-joints.
Two-point BVP studies the solution of a differential equation constrained to
the provided start and target conditions. Li et al. [27] approached motion
planning problems as a trajectory optimization problem for given initial and
final conditions. Xie et al. [28] combined optimal planning algorithm with a
two-point BVP framework to solve optimal motion planning problem.

As deep neural network approaches are introduced, there have been great
improvements in MIB tasks. Harvey et al. [8] demonstrated studio-quality MIB
methods based on their motion transition network [13]. Also, they leveraged
the least-squares generative adversarial network (LSGAN) [29] to make
generated motions to be more natural. However, these auto-regressive
approaches are prone to long-horizon generation as the error accumulates over
time and, thus, is difficult to parallelize. To overcome this, Kaufmann et al.
[9] employed a convolutional autoencoder by representing motion data in a
matrix that can be interpreted as an image and showed that a non-
autoregressive method could produce comparable results in MIB without
degrading visual results. In contrast to prior work, we propose a controllable
motion generation method on top of the MIB framework. The most relevant work
of our approach is Harvey et al. [8], which, in contrast, only performs MIB
tasks without controllability.

##  3 Problem Formulation

###  3.1 Motion Data Representation

We describe a human pose with joint positions and joint rotations. Joint
positions are expressed on a real-world scale. There are multiple ways to
represent rotations, such as Euler angles, axis-angle representation, and
quaternions. We compare the model’s performance on the three rotation
expressions in Table 1. In our experiment, the Euler angle representation is
highly unstable during training. Empirically, the quaternion format
outperforms other rotation expression formats in both performance and training
stability, so we choose quaternion vectors as our rotation representation as
previous work [8] did.

Length (frames) | 30 | 40 | 60 | 80 | 120
---|---|---|---|---|---
Euler | 2.17 | 2.93 | N/A | N/A | N/A
Axis-angle | 1.38 | 1.51 | 2.15 | 2.89 | 4.54
Quaternion | 1.19 | 1.35 | 1.98 | 2.70 | 4.25
Table 1: Motion in-betweening L2P performance comparison by different rotation
representations on the LAFAN1 dataset. “N/A” indicates the setting was not
able to train because of instability in the learning stage. Our CMIB model is
used for evaluation.

We represent a joint position vector $\mathbf{p}\in\mathbb{R}^{3}$ in
Euclidean space and adopt the quaternion $\mathbf{q}\in\mathbb{R}^{4}$ as a
rotational representation of each joint. Positions and rotations at specific
key-pose $t\in[1,T]$ for $J$ number of joints are denoted in a vectorized form
$P_{t}\in\mathbb{R}^{3J}$ and $R_{t}\in\mathbb{R}^{4J}$, respectively. All
positional and rotational values are represented in the global coordinate
system in this paper.

###  3.2 Conditional Motion In-betweening

A wide range of studies have been conducted to generate motion, but motion in-
betweening (MIB) has been less focused despite its practical usefulness in
industries. MIB can be viewed as a boundary value problem for motion synthesis
network $g$ that satisfies the initial conditions:

$g(t_{1})=\begin{bmatrix}P_{1},&R_{1}\end{bmatrix}\quad
g(t_{T})=\begin{bmatrix}P_{T},&R_{T}\end{bmatrix}$ (1)

Recent studies [8, 9] approached MIB as a problem conditioned only by both
ends, but we extend the condition not only both ends but also additional
intermediate frame and semantic information. In this work, we focus on two
conditioning aspects: pose-conditioned and semantic-conditioned MIB. First,
pose/conditioned motion in/betweening is a method of connecting starting and
target pose while satisfying a given anchor pose as well. The other constraint
is semantic information in motion. Motion data contains semantic information,
such as emotions or action contents. Motion stylization [30, 31] has covered
methods to transfer style information while preserving motion’s content
features. Unlike motion stylization, our semantic-conditioned MIB needs to
generate stylized content at once since it does not have available content
motion frames other than starting and target pose.

##  4 Proposed Method

We propose a conditional motion in-betweening (CMIB) method that performs
conditioned motion generation for both pose and semantic context, based on a
Transformer architecture. Figure 2 illustrates the overall architecture of the
proposed method.

Figure 2: Overview of the model architecture. During training, we dynamically
sample masking frames and replace those frames with interpolation from given
poses. Then, the interpolated sequence is fed to the backbone Transformer
encoder network with prepended semantic embedding. This task aims at
generating natural human motion from given poses while providing semantic
controllability.

###  4.1 Pose Interpolation

To perform MIB, starting, anchor (optional), and target poses should be
provided. Instead of initializing key-poses with arbitrary values, we
initialize our network with baseline interpolation to facilitate the learning
process. For the given poses, the joint coordinates are interpolated with
linear interpolation (Eq. 2), and joint rotations are interpolated by
spherical linear interpolation (Eq. 3) in quaternion. Starting and target
positions are denoted as
$\mathbf{p}_{\mathrm{start}},\mathbf{p}_{\mathrm{target}}$, and the position
of the anchor pose given at $k$-th key-pose is written as $\mathbf{p}_{k}$.
Joint rotations of start ($\mathbf{q}_{\mathrm{start}}$), target
($\mathbf{q}_{\mathrm{start}}$), and anchor ($\mathbf{q}_{k}$) poses are
represented in the same manner.

$\operatorname{LERP}(\mathbf{p}_{\mathrm{start}},\mathbf{p}_{\mathrm{target}},\mathbf{p}_{k},t)=\begin{cases}\frac{k-t}{k-1}\mathbf{p}_{\mathrm{start}}+\frac{t-1}{k-1}\mathbf{p}_{k}&\text{for
}1\leq t<k\\\
\frac{T-t}{T-k}\mathbf{p}_{k}+\frac{t-k}{T-k}\mathbf{p}_{\mathrm{target}}&\text{for
}k\leq t\leq T\end{cases}$ (2)
$\operatorname{SLERP}(\mathbf{q}_{\mathrm{start}},\mathbf{q}_{\mathrm{target}},\mathbf{q}_{k},t)=\begin{cases}\frac{\sin\left(\frac{k-t}{k-1}\theta_{1}\right)}{\sin\theta_{1}}\mathbf{q}_{\mathrm{start}}+\frac{\sin\left(\frac{t-1}{k-1}\theta_{1}\right)}{\sin\theta_{1}}\mathbf{q}_{k}&\text{for
}1\leq t<k\\\
\frac{\sin\left(\frac{T-t}{T-k}\theta_{2}\right)}{\sin\theta_{2}}\mathbf{q}_{k}+\frac{\sin\left(\frac{t-k}{T-k}\theta_{2}\right)}{\sin\theta_{2}}\mathbf{q}_{\mathrm{target}}&\text{for
}k\leq t\leq T\end{cases}$ (3)

where $\theta_{1}=\arccos(\mathbf{q}_{\mathrm{start}}\cdot\mathbf{q}_{k})$ and
$\theta_{2}=\arccos(\mathbf{q}_{k}\cdot\mathbf{q}_{\mathrm{target}})$.

###  4.2 Model Architecture

We utilize a Transformer encoder to implement CMIB. Since our objective is to
create natural motion between given key-poses, we make the Transformer’s
multi-head self-attention to attend to all given input poses without masking.
Input motion data is interpolated with LERP and SLERP before feeding into the
motion encoder. Input motion can be compactly represented by vectorized
positions $P_{t}\in\mathbb{R}^{3J}$ and rotations $R_{t}\in\mathbb{R}^{4J}$ at
time step $t$:

$X=\begin{bmatrix}P_{1},&R_{1}\\\ P_{2},&R_{2}\\\ P_{3},&R_{3}\\\
\vdots&\vdots\\\ P_{T},&R_{T}\end{bmatrix}=\begin{bmatrix}\mathbf{x}_{1}\\\
\mathbf{x}_{2}\\\ \mathbf{x}_{3}\\\ \vdots\\\
\mathbf{x}_{T}\end{bmatrix}\in\mathbb{R}^{T\times d}$ (4)

where $d=J\cdot(3+4)$.

Since the non-autoregressive Transformer is leveraged, additional temporal
information of the motion needs to be incorporated. In this work, we adopt
learned positional embedding [32]. The dimension of positional embedding is
set to be equal to that of pose representation $\mathbf{x}_{t}$. Learned
positional embedding is added before feeding into the Transformer. With this,
we can write the input motion matrix $I$:

$I=\begin{bmatrix}\mathbf{x}_{1}+PE_{1}\\\ \mathbf{x}_{2}+PE_{2}\\\
\mathbf{x}_{3}+PE_{3}\\\ \vdots\\\
\mathbf{x}_{T}+PE_{T}\end{bmatrix}\in\mathbb{R}^{T\times d}$ (5)

where $PE_{t}\in\mathbb{R}^{d},1\leq t\leq T$.

Our motion encoder uses multiple encoder layers consisting of multi-head
attention and position-wise feed-forward networks. Each single head attention
is computed as:

$\displaystyle Q$ $\displaystyle=W_{i}^{Q}I$ (6) $\displaystyle K$
$\displaystyle=W_{i}^{K}I$ (7) $\displaystyle V$ $\displaystyle=W_{i}^{V}I$
(8) $\displaystyle H_{i}$
$\displaystyle=\operatorname{softmax}\left(\frac{QK^{\top}}{\sqrt{d_{k}}}\right)V$
(9)

where $m$ is the number of heads, $d_{k}={\frac{d}{m}}$ is the dimension of
motion representation and $W_{i}^{Q},W_{i}^{K}$ and $W_{i}^{V}$ are trainable
parameter matrices for $i$-th head $H_{i}$. This single-head computation can
be extended to the multi-head version by concatenating multiple single-heads
followed by additional projection with $W^{O}$:

$\operatorname{MultiHead}(H_{1},\cdots,H_{m})=W^{O}\operatorname{Concat}(H_{1},\cdots,H_{m})$
(10)

where $W^{O}\in\mathbb{R}^{md\times d}$. A fully connected feed-forward
network (FCN) projects the encoder’s representation dimension to the motion
representation dimension. FCN block is composed of two stacked linear
transformations with GeLU activation [33]. Residual connection [34] is used
for both multi-head attention and FCNs followed by layer normalization. The
Transformer encoder outputs a matrix of the same size as $I$, representing the
complete predicted motion sequence.

Since our model learns positions and rotations in the global coordinate
system, link lengths from the predicted output are not guaranteed to have the
same lengths as the given kinematic chain. To make the link lengths
consistent, we linearly scale the predicted link lengths to have the same
lengths as the predefined kinematic chain.

###  4.3 Training Phase

####  4.3.1 Randomized Shuffled Anchor Pose

Pose conditioning is a way of generating motion to make it satisfy a given
anchor pose during the MIB process. An anchor pose can be located between the
start and target poses and it should be physically feasible in the context of
MIB. This makes a clear difference from existing MIB methods, which perform
MIB only for starting and target poses. Inspired from dynamic masking in
RoBERTa [35], we uniformly sample the anchor key-pose from training motion
sequences for every iteration. The sampled anchor pose is provided to the
Transformer encoder along with the start and target poses after pose
interpolation.

####  4.3.2 Motion Data Augmentation

The quality of pose-conditioned motion in-betweening is affected by the motion
path distribution of the training dataset. However, training motion data are
often insufficient to fully cover various human motions. In the image
processing domain, a number of data augmentation methods (e.g., random
cropping or warping) are widely used, however, most of the existing
augmentation methods in vision domains are not directly applicable to the
motion domain due to their temporal characteristics. Harvey et al. [13] used
simple augmentation strategies of mirroring the motion data in the forward
direction of the character. However, this method has a clear limitation in
that it is not able to create a qualitatively different motion. Here, we
introduce a motion augmentation method by first defining a probability
distribution of smooth trajectories interpolating the start and target points.
This method generates multiple motion trajectories from existing motions
utilizing Gaussian Random Path (GRP) [10].

Suppose that $(x_{t},y_{t})\in\mathbb{R}^{2}$ be a two-dimensional root joint
position projected onto the ground plane extracted from given three-
dimensional position vector $p_{t}\in\mathbb{R}^{3}$ and
$(\mathbf{x}_{\mathcal{P}},\mathbf{y}_{\mathcal{P}})=\left\\{(x_{t},y_{t})\,|\,t=1,\dots,T\right\\}$
be a two-dimensional trajectory of length $T$. Let
$\left\\{\mathbf{x}_{a},\mathbf{y}_{a}\right\\}=\left\\{(x_{1},y_{1}),(x_{T},y_{T})\right\\}$
be a pair of start and target anchoring root positions. Our proposed
augmentation method utilizes $\mathbf{x}_{\mathcal{P}}$ and anchoring root
vectors $\left\\{\mathbf{x}_{a},\mathbf{y}_{a}\right\\}$ to augment new root
path $\tilde{\textbf{y}}_{\mathcal{P}}$. Note that we assume the monotonicity
of $\mathbf{x}_{\mathcal{P}}$. Given a squared exponential kernel function
$\mathbf{k}(\mathbf{x},\mathbf{x}^{\prime})=\exp(-(\mathbf{x}-\mathbf{x}^{\prime})^{2}/{2})$
and $\mathbf{x}_{\mathcal{P}}$, a Gaussian random path distribution
$\tilde{\mathbf{y}}_{\mathcal{P}}$ is specified with a mean path
$\mu_{\mathcal{P}}$ and a covariance matrix $K_{\mathcal{P}}$ as following:

$\tilde{\textbf{y}}_{\mathcal{P}}\sim\mathcal{N}(\mu_{\mathcal{P}},K_{\mathcal{P}}),$
(11)

where

$\displaystyle\mu_{\mathcal{P}}$
$\displaystyle=\textbf{k}(\textbf{x}_{\mathcal{P}},\textbf{x}_{a})(\textbf{K}_{a}+\sigma^{2}_{w}I)^{-1}\textbf{y}_{a},$
(12) $\displaystyle K_{\mathcal{P}}$
$\displaystyle=\textbf{K}_{\mathcal{P}}-\textbf{k}(\textbf{x}_{\mathcal{P}},\textbf{x}_{a})(\textbf{K}_{a}+\sigma^{2}_{w}I)^{-1}\textbf{k}(\textbf{x}_{\mathcal{P}},\textbf{x}_{a})^{T},$
(13)

$\mathbf{k}(\mathbf{x}_{\mathcal{P}},\mathbf{x}_{a})\in\mathbb{R}^{T\times 2}$
is a kernel matrix of $\mathbf{x}$’s path indices and anchored start and
target. The kernel matrix of start and target indices is represented as
$\mathbf{K}_{a}=\mathbf{K}(\mathbf{x}_{a},\mathbf{x}_{a})\in\mathbb{R}^{2\times
2}$, and
$\mathbf{K}_{\mathcal{P}}=\mathbf{K}(\mathbf{x}_{\mathcal{P}},\mathbf{x}_{\mathcal{P}})\in\mathbb{R}^{T\times
T}$ indicates a kernel matrix of given $\mathbf{x}_{\textrm{path}}$. In the
sense of Cholesky decomposition of $\mathbf{LL}^{T}=K_{\mathcal{P}}$, Eq. 11
is equivalent to
$\tilde{\mathbf{y}}_{\mathcal{P}}=\mu_{\mathcal{P}}+\mathbf{Lu}$, where
$\mathbf{u}\sim\mathcal{N}(0,I)$.

In order to augment motion trajectory from ground truth motion, we first
rotate the original trajectory to make starting and target positions aligned
on the $X$-axis to facilitate the GRP process. In the rotated system, the
proposed motion augmentation process samples a new trajectory, which has the
same $\mathbf{x}$ values but new $\tilde{\mathbf{y}}$ values from the GRP
process. We have fixed the progress on $X$ and $Z$ axes since motion should
maintain its starting and target position and changing the $Z$-axis
empirically yields unnatural motion sequences. Therefore, motion augmentation
is applied only for ‘walking’ and ‘running’ motions. Through the augmentation
process, the rotated path
$(\mathbf{x}_{\mathcal{P}},\mathbf{y}_{\mathcal{P}},\mathbf{z}_{\mathcal{P}})$
can be augmented to multiple sampled paths
$(\mathbf{x}_{\mathcal{P}},\tilde{\mathbf{y}}_{\mathcal{P}},\mathbf{z}_{\mathcal{P}})$.
On top of the positional augmentation, we further compute the orientation
difference between the original and augmented trajectories in the $XY$ plane
and rotate the augmented sequence accordingly on the $XY$-plane to make the
generated motion face the correct direction as the original motion. Figure 3
illustrates the process of the proposed motion augmentation method.

(a)

(b)

(c)

(d)

Figure 3: The original trajectory is rotated to align the $+X$-axis
(character’s front direction), then GRP is employed to sample a trajectory.
The generated trajectory is rotated back to match the original trajectory’s
starting and target position. The root joint’s trajectory is visualized.

####  4.3.3 Semantic Embedding

Motion representation and semantic information need to be learned together to
achieve semantic-conditioned motion in-betweening. To this end, we introduce
semantic embedding, which projects motion’s high-level semantics into a look-
up table. We prepend the semantic embedding to be the first token of the
motion input matrix so that the Transformer encoder can attend to this
information:

$I_{\mathrm{semantic}}=\operatorname{Concat}(\mathbf{s},I)\in\mathbb{R}^{(T+1)\times
d}$ (14)

where $\mathbf{s}$ indicates the semantic embedding. We use the extended input
motion representation before feeding the Transformer encoder.

###  4.4 Loss

There are three forms of learning losses in our model: semantic loss, position
loss, and rotation loss.

First, semantic loss is a reconstruction loss. The first predicted token is a
special token to represent semantics in motion. We make the model reconstruct
the first token for every sequence:

$\mathcal{L}_{\mathrm{semantic}}=\frac{1}{N}\sum_{N=1}^{N}\lVert\mathbf{s}-\mathbf{\hat{s}}\rVert_{1}$
(15)

where $N$ is the total number of data and $\mathbf{s}$ means semantic
embedding.

Position and rotation losses are the average L1 distance between predicted
sequences and their ground truth values:

$\displaystyle\mathcal{L}_{\mathrm{position}}$
$\displaystyle=\frac{1}{N}\frac{1}{T}\sum_{N=1}^{N}\sum_{t=1}^{T}\lVert\mathbf{p}_{t}-\mathbf{\hat{p}}_{t}\rVert_{1}$
(16) $\displaystyle\mathcal{L}_{\mathrm{rotation}}$
$\displaystyle=\frac{1}{N}\frac{1}{T}\sum_{N=1}^{N}\sum_{t=1}^{T}\lVert\mathbf{q}_{t}-\mathbf{\hat{q}}_{t}\rVert_{1}$
(17)

Our total loss is defined as:

$\mathcal{L}_{\mathrm{total}}=w_{1}\mathcal{L}_{\mathrm{semantic}}+w_{2}\mathcal{L}_{\mathrm{position}}+w_{3}\mathcal{L}_{\mathrm{rotation}}$
(18)

All of our losses are scaled to be approximately equal before training begins.

###  4.5 Hyperparameters

Model is trained with AdamW [36] optimizer with a learning rate of 0.0001,
$\beta_{1}=0.9$, $\beta_{2}=0.99$, and weight decay of 0.01. Also, we set
Transformer encoder to have 7 heads, 8 encoder layers, 2,048 feedforward
dimensions, and 0.05 dropout probability during training. Weights for the
total losses $w_{1},w_{2}$, and $w_{3}$ are set to 1.5, 0.05, and 2.0
respectively. In all our experiments, minibatch size is set to $32$.

##  5 Experiments

This section demonstrates the experimental settings and presents qualitative
and quantitative results. Please refer to our project web
page111https://jihoonerd.github.io/Conditional-Motion-In-Betweening for more
videos from the experiments.

###  5.1 Datasets

We evaluate our algorithm on four widely-used motion datasets for motion in-
betweening (MIB) task: LAFAN1 [8], HumanEva [37], HUMAN4D [38], and MPI-HDM05
[39]. We split the training and test datasets by subjects which is described
in Table 2.

LAFAN1 [8]: LAFAN1 is a high-quality public motion capture dataset. It
contains $15$ actions such as walking, dancing, fighting, jumping, etc, with
approximately $4.6$ hours long and contains $496,672$ frames taken by five
actors. It was used in Harvey et al. [8], which first proposed a robust MIB
method. We evaluate the performance of our proposed method using this dataset
for both unconstrained MIB and CMIB tasks.

HumanEva [37]: This dataset is obtained from a 3D whole-body motion capture
system. It is recorded by four subjects with six common actions such as
walking, jogging, gesturing, etc. Three RGB cameras and four grayscale cameras
are used for the recording.

HUMAN4D [38]: This is a recently-collected large and multimodal public motion
dataset with various human actions. This dataset is captured from $24$ MoCap
cameras with four depth sensors. The full dataset is composed of $50,306$
frames with $19$ activity labels.

MPI-HDM05 [39]: MPI-HDM05 is a research-purpose dataset especially for motion
analysis, synthesis, and classification. It contains more than three hours of
recorded motion frames.

Table 2: Experiment configuration of train and test data. Dataset | Train (subject) | Test (subject)
---|---|---
LAFAN1 | 1, 2, 3, 4 | 5
HumanEva | 1, 2 | 3
HUMAN4D | 1, 2, 3, 4, 5, 6, 7 | 8
MPI-HDM05 | bk, dg, mm | tr

###  5.2 Motion Data Pre-processing

We adapt the same pre-processing procedures with [8] for a fair comparison. In
particular, we split each motion into a fixed-width window where all windowed
motion sub-sequences are rotated to align $X^{+}$ axis ahead at the $10$-th
frame. For semantic conditioning, we choose labeled actions as semantic
information of the motion. Positions and rotations are represented in the
global coordinate frame. We also build a deploy pipeline for the Unity
Engine222https://unity.com to present qualitative results.

###  5.3 Motion In-betweening Results

We use L2P and L2Q used in [8] to evaluate our model’s MIB performance. L2P
and L2Q are the average L2 norm of global positions and quaternions:

$\displaystyle\mathrm{L2P}=\frac{1}{N}\frac{1}{T}\sum_{N=1}^{N}\sum_{t=1}^{T}\lVert\mathbf{p}_{t}-\hat{\mathbf{p}}_{t}\rVert_{2}$
(19)
$\displaystyle\mathrm{L2Q}=\frac{1}{N}\frac{1}{T}\sum_{N=1}^{N}\sum_{t=1}^{T}\lVert\mathbf{q}_{t}-\hat{\mathbf{q}}_{t}\rVert_{2}$
(20)

where $N$ indicates the number of data being evaluated. Note that global
positions are normalized before calculating L2P.

Here, we compare our model with the state-of-the-art robust motion in-
betweening models in MIB tasks. Since the implementation of RMIB [8] and SSMCT
[40] are not publicly available, the quantitative results of both models that
are not reported in the original paper are provided by our own reproduced
implementation. We would like to note that our reproduced implementation also
shows similar results in their reported performance.

Table 3: Motion in-betweening results on LAFAN1.

Length (frames) | 30 | 40 | 60 | 80 | 120
---|---|---|---|---|---
Zero-Velocity | 6.60 | 8.44 | 10.20 | 13.42 | 18.69
Interp. | 2.32 | 3.05 | 3.87 | 5.63 | 9.12
Bi-LSTM | 1.56 | 2.13 | 3.54 | 4.80 | 8.91
ERD-QV [8] | 1.28 | 2.03 | 3.28 | 4.51 | 7.61
CAE [9] | 1.23 | 1.64 | 2.68 | 3.46 | 5.43
SSMCT [40] | 1.10 | 1.33 | 2.04 | 2.78 | 4.32
CMIB (Ours) | 1.19 | 1.35 | 1.98 | 2.70 | 4.25
(a) L2P

Length (frames) | 30 | 40 | 60 | 80 | 120
---|---|---|---|---|---
Zero-Velocity | 1.51 | 1.72 | 1.91 | 2.22 | 2.67
Interp. | 0.98 | 1.15 | 1.34 | 1.72 | 2.28
Bi-LSTM | 0.79 | 0.88 | 1.25 | 1.52 | 2.07
ERD-QV [8] | 0.69 | 0.79 | 1.12 | 1.48 | 1.51
CAE [9] | 0.84 | 0.95 | 1.30 | 1.65 | 2.10
SSMCT [40] | 0.61 | 0.76 | 1.14 | 1.53 | 1.98
CMIB (Ours) | 0.59 | 0.71 | 1.04 | 1.42 | 1.90
(b) L2Q

Our model is compared with zero-velocity, interpolation, bidirectional LSTM
model, ERD-QV [8], CAE [9], and SSMCT [40]. The zero-velocity baseline
interpolates the missing frames with the latest frame. The interpolation
baseline model is the same as our baseline pose interpolation, which uses LERP
for joints’ positional values and SLERP for rotational (quaternion) values.
Table 3 shows the MIB evaluation results on the test dataset of LAFAN1, and
Table 4 presents L2P and L2Q score comparison on other public motion datasets.
Our proposed method presents comparable or better performances in most MIB
tasks, even with the additional controllability. Synthesized motions are
visualized in Figure 5.

Table 4: Motion in-betweening results on other motion datasets. Length (frames) | 40 | 60 | 80
---|---|---|---
Metric | L2P | L2Q | L2P | L2Q | L2P | L2Q
HumanEva [37] | Bi-LSTM | 3.71 | 0.80 | 4.51 | 1.30 | 4.92 | 1.57
ERD-QV [8] | 3.58 | 0.73 | 4.32 | 1.26 | 4.73 | 1.42
CAE [9] | 3.49 | 0.83 | 3.98 | 1.42 | 4.65 | 1.73
SSMCT [40] | 3.42 | 0.73 | 3.88 | 0.85 | 4.52 | 0.96
CMIB (Ours) | 3.45 | 0.70 | 3.84 | 0.80 | 4.48 | 0.87
HUMAN4D [38] | Bi-LSTM | 3.48 | 0.83 | 5.11 | 1.08 | 6.23 | 1.24
ERD-QV [8] | 3.40 | 0.79 | 4.95 | 0.99 | 5.93 | 1.15
CAE [9] | 3.29 | 0.87 | 4.78 | 1.13 | 5.34 | 1.35
SSMCT [40] | 3.11 | 0.71 | 4.10 | 1.02 | 4.80 | 1.27
CMIB (Ours) | 3.10 | 0.70 | 4.02 | 0.95 | 4.66 | 1.17
MPI-HDM05 [39] | Bi-LSTM | 2.65 | 0.86 | 3.80 | 1.23 | 4.72 | 1.40
ERD-QV [8] | 2.60 | 0.82 | 3.63 | 1.08 | 4.60 | 1.26
CAE [9] | 2.54 | 0.92 | 3.58 | 1.39 | 4.41 | 1.52
SSMCT [40] | 2.33 | 0.85 | 3.18 | 1.21 | 3.85 | 1.49
CMIB (Ours) | 2.30 | 0.81 | 3.02 | 1.15 | 3.78 | 1.45

Figure 4 presents the computation time of different batch sizes and in-
betweening frames. This supports that the non-autoregressive model has a great
advantage in inference time speed performance over the autoregressive models.

(a) In-betweening length: 30 frames

(b) In-betweening length: 80 frames

Figure 4: Speed performance. CPU inference time is measured with Intel® Xeon®
Gold 5120 CPU @ 2.20GHz. Inference time is measured as an average after 30
trials.

Figure 5: Unconstrained in-betweening results. Intermediate poses are
generated from starting pose (red) to target pose (blue).

###  5.4 Pose-conditioned Motion In-betweening Results

Pose-conditioned MIB is a process of generating natural skeletal motion with
the start, target, and additional anchor poses constraints which can be
thought of as having additional anchor pose constraints to the unconstrained
MIB task. Since a non-autoregressive Transformer architecture of utilized, we
can leverage multiple numbers of anchor poses while training. In this paper,
however, we simply assume that a single anchor pose is given for the pose-
conditioned MIB task, and while training, an anchor pose is uniformly sampled
within the horizon of a motion sequence (i.e., $t\sim\mathcal{U}(1,T)$ where
$T$ is the sequence length).

Figure 6: (Top) Unconstrained MIB result from the start pose (red) to the
target pose (blue). (Bottom) Pose-conditioned MIB result with an additional
anchor pose (green)

Note that the pose-conditioned MIB results are subjected to the diversity of
trajectories within the dataset, where we observe that pose-conditioned
generation often fails when the anchor pose is located outside the plausible
reason. However, this limitation can be alleviated via the proposed motion
data augmentation method in Section 4.3.2. To validate the effectiveness of
the proposed motion data augmentation on the pose-conditioned MIB tasks, we
select the poses at $t=20,\,40,\,\text{and}\,60$ on aiming, dance, fight, run,
and walk motions of LAFAN1 (total $670$ motions). Then, we add random
perturbations to the $(x,\,y)$ position of the anchor pose and check how the
generated sequences pass through the given anchor pose in the $XY$ plane. We
further change the orientation of the start and target poses to match the
given anchor pose.

Table 5 shows the comparative results of the pose-conditioned MIB task with
and without the proposed motion augmentation method in terms of the L2 norm
between the given and generated root positions at
$t=20,\,40,\,\text{and}~{}60$. Our proposed augmentation method greatly
improves the pose-conditioned MIB performance on all anchor positions. Figure
6 depicts the pose-conditioned MIB result. When given a jumping anchor pose
(green), the entire generated motion changes accordingly to make the jump
motion.

Table 5: Ablation study on augmentation. L2 distance at anchor pose. Anchor Position | $t=20$ | $t=40$ | $t=60$
---|---|---|---
CMIB (w/o Aug) | 0.35 | 0.78 | 0.35
CMIB (w/ Aug) | 0.18 | 0.18 | 0.19

###  5.5 Semantic-conditioned Motion In-betweening Results

In this experiment, we set conditioning semantics as $15$ action labels
provided from the LAFAN1 dataset. We evaluate our model based on the
reconstruction error of the semantic token values. For quantitative
evaluation, we divide the test dataset by action labels and make the model
conduct MIB for available action labels. Pose conditioning is disabled for the
experiments (i.e., no anchor pose is given). Here we evaluate on the model’s
performance in terms of L2P. We assume that the corresponding ground truth
labels should yield the smallest error. Table 6 describes the results and
Figure 7 visualizes the results of semantic-conditioned MIB.

From the visualized results, we found that infilling horizon affects the
visual quality of semantic-conditioned MIB. Long-horizon MIB tends to produce
better results compared to the short-horizon task in the semantic-conditioned
generation. For example, in the case of jumping motions, it is difficult to
generate a complete jumping motion with a 30-frames horizon, but long-horizon
infilling is more likely to produce clear jumping motions. This suggests that
there will be a minimum number of frames to reflect the semantics of motion.

Table 6: Semantic-conditioned motion in-betweening results. Column-wise
direction indicates conditioning semantics provided to model and row-wise
direction means ground truth conditions in the test dataset.

| walk | run | dance | jump | fight
---|---|---|---|---|---
walk | 1.03 | 1.02 | 1.26 | 1.15 | 1.11
run | 1.35 | 1.31 | 1.58 | 1.60 | 1.37
dance | 1.64 | 1.63 | 1.69 | 1.66 | 1.64
jump | 1.64 | 1.67 | 1.95 | 1.34 | 1.73
fight | 1.82 | 1.81 | 2.11 | 1.89 | 1.82
(a) L2P: 40 Frames

| walk | run | dance | jump | fight
---|---|---|---|---|---
walk | 1.65 | 1.77 | 1.98 | 1.87 | 1.77
run | 2.00 | 1.92 | 2.29 | 2.21 | 2.00
dance | 2.02 | 2.04 | 2.14 | 2.01 | 2.08
jump | 2.12 | 2.20 | 2.38 | 1.83 | 2.22
fight | 2.51 | 2.49 | 2.84 | 2.58 | 2.50
(b) L2P: 60 Frames

| walk | run | dance | jump | fight
---|---|---|---|---|---
walk | 2.43 | 2.50 | 2.74 | 2.60 | 2.52
run | 3.13 | 2.92 | 3.40 | 3.29 | 2.97
dance | 2.42 | 2.43 | 2.52 | 2.37 | 2.46
jump | 2.60 | 2.82 | 2.87 | 2.38 | 2.78
fight | 3.40 | 3.44 | 3.67 | 3.50 | 3.34
(c) L2P: 80 Frames

| walk | run | dance | jump | fight
---|---|---|---|---|---
walk | 4.43 | 4.45 | 4.76 | 4.79 | 4.81
run | 5.98 | 5.58 | 6.40 | 6.55 | 6.27
dance | 3.68 | 3.89 | 3.76 | 3.69 | 3.93
jump | 3.63 | 4.59 | 3.84 | 3.43 | 3.86
fight | 5.02 | 5.10 | 5.21 | 5.14 | 4.86
(d) L2P: 120 Frames

(a)

(b)

Figure 7: Semantic-conditioned in-betweening results. Conditioning semantics
are walk (top), dance (middle), and jump (bottom). The left column depicts the
entire sequence trajectory with starting (red) and target (blue) poses. The
right column visualizes interim three consecutive poses for visual comparison
between different semantics.

##  6 Limitation and Discussion

In this work, we confirm that the CMIB model can perform the most of in-
betweening settings, but there are some limitations. First, generated motion
does not take volume into consideration, which produces motion with
penetration between the human body as visualized in Figure 8. It can be
mitigated by including a post-processing pipeline or integrating loss term for
penetrating configuration as a regularizer. The other failure case is
conditioning failure. This case is similar to mode collapse in GAN,
synthesizing nearly identical motion regardless of conditioning semantics. We
suspect that the problem comes from two objectives that are difficult to
satisfy simultaneously. For instance, if given starting and target poses are
too far apart, motion in-betweening objective mandates a model to generate
running motion without regard to given semantics such as ‘fight’ or ‘dance’.
This is depicted in Figure 9. Empirically, conditioned generation performs
best when given starting and target poses are feasible enough to generate the
given condition.

Figure 8: Motion in-betweening results with a penetration. Figure 9: Failure
case for semantic control. Grey, orange, green, and purple are conditionally
generated from ‘walk’, ‘jump’, ‘fight’, and ‘dance’ labels, respectively. Red
and blue poses are starting and target poses.

##  7 Conclusion

In this study, we have focused on the problem of adding controllability to
motion in-betweening (MIB) tasks. In particular, we proposed a conditional
motion in-betweening (CMIB) method using non-autoregressive Transformer
encoder architecture where pose-conditioned and semantic-conditioned MIB
methods are achieved via randomized shuffled anchor pose and semantic
embedding token, respectively. An effective motion data augmentation method is
also presented to increase the performance of the pose-conditioned MIB. Our
proposed method outperforms existing state-of-the-art MIB with respect to pose
prediction accuracy and inference time while providing additional
controllability.

While our approach demonstrates controllability in MIB, there are still
promising areas where more research is required. As our model relies on the
non-autoregressive structure, the horizon it predicts is limited to the
predefined maximum length, and the evaluation metric is not suitable for
expressing the diversity of human motions. It would be interesting to combine
our method with explicit density estimation methods to better capture the
possible human motion distributions. Furthermore, we will continue to extend
controllable motion generation combined with other modalities such as natural
language commands.

## Acknowledgements

This work was supported by Institute of Information & communications
Technology Planning & Evaluation (IITP) grant funded by the Korea
government(MSIT) (No. 2019-0-00079 , Artificial Intelligence Graduate School
Program(Korea University)).

## References

  * [1] J. Zhao, N. I. Badler, Inverse kinematics positioning using nonlinear programming for highly articulated figures, ACM Transactions on Graphics (TOG) 13 (4) (1994) 313–336.
  * [2] Z. Xiao, H. Nait-Charif, J. J. Zhang, Automatic estimation of skeletal motion from optical motion capture data, in: International Workshop on Motion in Games, Springer, 2008, pp. 144–153.
  * [3] K. Fragkiadaki, S. Levine, P. Felsen, J. Malik, Recurrent network models for human dynamics, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 4346–4354.
  * [4] P. Ghosh, J. Song, E. Aksan, O. Hilliges, Learning human motion models for long-term predictions, in: 2017 International Conference on 3D Vision (3DV), IEEE, 2017, pp. 458–466.
  * [5] J. Won, K. Lee, C. O’Sullivan, J. K. Hodgins, J. Lee, Generating and ranking diverse multi-character interactions, ACM Transactions on Graphics (TOG) 33 (6) (2014) 1–12.
  * [6] H. Ahn, T. Ha, Y. Choi, H. Yoo, S. Oh, Text2action: Generative adversarial synthesis from language to action, in: 2018 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2018, pp. 5915–5920.
  * [7] C. Guo, X. Zuo, S. Wang, S. Zou, Q. Sun, A. Deng, M. Gong, L. Cheng, Action2motion: Conditioned generation of 3d human motions, in: Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 2021–2029.
  * [8] F. G. Harvey, M. Yurick, D. Nowrouzezahrai, C. Pal, Robust motion in-betweening, ACM Transactions on Graphics (TOG) 39 (4) (2020) 60–1.
  * [9] M. Kaufmann, E. Aksan, J. Song, F. Pece, R. Ziegler, O. Hilliges, Convolutional autoencoders for human motion infilling, in: 2020 International Conference on 3D Vision (3DV), IEEE, 2020, pp. 918–927.
  * [10] S. Choi, K. Lee, S. Oh, Gaussian random paths for real-time motion planning, in: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2016, pp. 1456–1461.
  * [11] J. Martinez, M. J. Black, J. Romero, On human motion prediction using recurrent neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2891–2900.
  * [12] L.-Y. Gui, Y.-X. Wang, X. Liang, J. M. Moura, Adversarial geometry-aware human motion prediction, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 786–803.
  * [13] F. G. Harvey, C. Pal, Recurrent transition networks for character locomotion, in: SIGGRAPH Asia 2018 Technical Briefs, 2018, pp. 1–4.
  * [14] E. Aksan, M. Kaufmann, O. Hilliges, Structured prediction helps 3d human motion modelling, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 7144–7153.
  * [15] L. Rossi, M. Paolanti, R. Pierdicca, E. Frontoni, Human trajectory prediction and generation using lstm models and gans, Pattern Recognition 120 (2021) 108136\.
  * [16] E. Aksan, P. Cao, M. Kaufmann, O. Hilliges, A spatio-temporal transformer for 3d human motion prediction, arXiv preprint arXiv:2004.08692 (2020).
  * [17] A. Martínez-González, M. Villamizar, J.-M. Odobez, Pose transformers (potr): Human motion prediction with non-autoregressive transformers, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2276–2284.
  * [18] G. E. Henter, S. Alexanderson, J. Beskow, MoGlow: Probabilistic and controllable motion synthesis using normalising flows, ACM Transactions on Graphics 39 (4) (2020) 236:1–236:14.  doi:10.1145/3414685.3417836.
  * [19] H. Y. Ling, F. Zinno, G. Cheng, M. Van De Panne, Character controllers using motion vaes, ACM Transactions on Graphics (TOG) 39 (4) (2020) 40–1.
  * [20] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, A. A. Efros, Context encoders: Feature learning by inpainting, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2536–2544.
  * [21] Y. Zeng, J. Fu, H. Chao, Learning joint spatial-temporal transformations for video inpainting, in: European Conference on Computer Vision, Springer, 2020, pp. 528–543.
  * [22] R. M. Escher, R. A. de Bem, P. L. J. Drews, Fast spatial-temporal transformer network, in: 2021 34th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), IEEE, 2021, pp. 65–72.
  * [23] W. Bao, W.-S. Lai, C. Ma, X. Zhang, Z. Gao, M.-H. Yang, Depth-aware video frame interpolation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3703–3712.
  * [24] C. Rose, M. F. Cohen, B. Bodenheimer, Verbs and adverbs: Multidimensional motion interpolation, IEEE Computer Graphics and Applications 18 (5) (1998) 32–40.
  * [25] T. Mukai, S. Kuriyama, Geostatistical motion interpolation, in: ACM SIGGRAPH 2005 Papers, 2005, pp. 1062–1070.
  * [26] A. M. Lehrmann, P. V. Gehler, S. Nowozin, Efficient nonlinear markov models for human motion, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1314–1321.
  * [27] H. Li, C.-H. Chen, G. Lakhani, Solving two-point boundary value problems in trajectory formation, in: IJCNN-91-Seattle International Joint Conference on Neural Networks, Vol. 1, IEEE, 1991, pp. 165–170.
  * [28] C. Xie, J. van den Berg, S. Patil, P. Abbeel, Toward asymptotically optimal motion planning for kinodynamic systems using a two-point boundary value problem solver, in: 2015 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2015, pp. 4187–4194.
  * [29] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, S. Paul Smolley, Least squares generative adversarial networks, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2794–2802.
  * [30] D. Holden, I. Habibie, I. Kusajima, T. Komura, Fast neural style transfer for motion data, IEEE computer graphics and applications 37 (4) (2017) 42–49.
  * [31] K. Aberman, Y. Weng, D. Lischinski, D. Cohen-Or, B. Chen, Unpaired motion style transfer from video to animation, ACM Transactions on Graphics (TOG) 39 (4) (2020) 64.
  * [32] J. Gehring, M. Auli, D. Grangier, D. Yarats, Y. N. Dauphin, Convolutional sequence to sequence learning, in: International Conference on Machine Learning, PMLR, 2017, pp. 1243–1252.
  * [33] D. Hendrycks, K. Gimpel, Gaussian error linear units (gelus), arXiv preprint arXiv:1606.08415 (2016).
  * [34] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [35] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019).
  * [36] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101 (2017).
  * [37] L. Sigal, A. O. Balan, M. J. Black, Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion, International journal of computer vision 87 (1-2) (2010) 4.
  * [38] A. Chatzitofis, L. Saroglou, P. Boutis, P. Drakoulis, N. Zioulis, S. Subramanyam, B. Kevelham, C. Charbonnier, P. Cesar, D. Zarpalas, et al., Human4d: A human-centric multimodal dataset for motions and immersive media, IEEE Access 8 (2020) 176241–176262.
  * [39] M. Müller, T. Röder, M. Clausen, B. Eberhardt, B. Krüger, A. Weber, Documentation mocap database hdm05, Tech. Rep. CG-2007-2, Universität Bonn (June 2007).
  * [40] Y. Duan, T. Shi, Z. Zou, Y. Lin, Z. Qian, B. Zhang, Y. Yuan, Single-shot motion completion with transformer, arXiv preprint arXiv:2103.00776 (2021).