# Adiabatic waveforms from extreme-mass-ratio inspirals: an analytical
approach

Soichiro Isoyama  School of Mathematics and STAG Research Centre, University
of Southampton, Southampton SO17 1BJ, United Kingdom  International Institute
of Physics, Universidade Federal do Rio Grande do Norte, Campus Universitário,
Lagoa Nova, Natal-RN 59078-970, Brazil  Ryuichi Fujita  Institute of Liberal
Arts, Otemon Gakuin University, Osaka 567-8502, Japan  Center for
Gravitational Physics, Yukawa Institute for Theoretical Physics, Kyoto
University, Kyoto 606-8502, Japan  Alvin J. K. Chua  Theoretical Astrophysics
Group, California Institute of Technology, Pasadena, CA 91125, USA  Hiroyuki
Nakano  Faculty of Law, Ryukoku University, Kyoto 612-8577, Japan  Adam Pound
School of Mathematics and STAG Research Centre, University of Southampton,
Southampton SO17 1BJ, United Kingdom  Norichika Sago  Department of Physics,
Kyoto University, Kyoto 606-8502, Japan  Advanced Mathematical Institute,
Osaka City University, Osaka 558-8585, Japan

###### Abstract

Scientific analysis for the gravitational-wave detector LISA will require
theoretical waveforms from extreme-mass-ratio inspirals (EMRIs) that
extensively cover all possible orbital and spin configurations around
astrophysical Kerr black holes. However, on-the-fly calculations of these
waveforms have not yet overcome the high dimensionality of the parameter
space. To confront this challenge, we present a user-ready EMRI waveform model
for generic (eccentric and inclined) orbits in Kerr spacetime, using an
analytical self-force approach. Our model accurately covers all EMRIs with
arbitrary inclination and black hole spin, up to modest eccentricity
($\lesssim 0.3$) and separation ($\gtrsim 2$–$10M$ from the last stable
orbit). In that regime, our waveforms are accurate at the leading ‘adiabatic’
order, and they approximately capture transient self-force resonances that
significantly impact the gravitational-wave phase. The model fills an urgent
need for extensive waveforms in ongoing data-analysis studies, and its
individual components will continue to be useful in future science-adequate
waveforms.

††preprint: YITP-21-114, KUNS-2901, OCU-PHYS-551, AP-GR-175

Introduction.—Gravitational-wave (GW) astronomy has revealed a cosmos brimming
with black holes (BHs) [1, 2, 3, 4], and as GW detectors improve, we will
continue to learn more about BHs’ properties and demographics. In the 2030s,
GW detectors in space (LISA, the Laser Interferometer Space Antenna [5], as
well as DECIGO [6], TianQin, and Taiji [7, 8, 9]) will observe unparalleled
probes of BHs: the inspirals of stellar-mass objects into supermassive BHs in
galactic cores [10, 11]. The waveforms from these extreme-mass-ratio inspirals
(EMRIs, with mass ratios $\eta\sim 10^{-4}$–$10^{-7}$) will contain a unique
wealth of information about the spacetime geometry of BHs, strong-field
physics in their vicinity, and the astrophysics of their stellar environments
[12, 13, 14, 15, 16].

The scientific potential of EMRIs has motivated the community to solve the
relativistic two-body problem in the small-mass-ratio regime, making use of
gravitational self-force (GSF) theory [17, 18, 19, 20, 21, 22, 23]: a
perturbative method in which the small body perturbs the central BH’s
spacetime, and the perturbation drives the body away from geodesic motion. GSF
theory has flourished over the past 25 years [24, 25, 26, 27, 28], yielding a
range of powerful tools for modelling EMRIs [29, 30, 31, 32, 33, 34, 35, 36].
The key goal of the EMRI modelling program is to generate waveforms for
generic (eccentric and inclined) inspiraling orbits about astrophysical Kerr
BHs, accounting for GSF effects. In order to enable accurate extraction of
EMRI parameters from a signal, a GSF model must ultimately be accurate to the
first subleading (‘first post-adiabatic’) order in $\eta$ [37, 38, 39, 40, 41,
42]. Attaining such accuracy will depend crucially upon calculations at
leading order, which can be used to produce adiabatic waveforms [43, 44, 45,
46, 47, 48] as a baseline in present-day data-analysis studies and for future
science-adequate waveforms.

After decades of progress [49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61,
62, 63, 64], adiabatic waveforms for generic Kerr orbits were obtained in 2021
[65]. Still, very few of these (mostly equatorial or spherical orbits) have
been simulated so far, and the raw techniques used in that work are unsuitable
in the provision of on-the-fly waveforms for data-analysis studies. The main
challenge is the high-dimensional parameter space of generic EMRIs. An
adiabatic evolution must compute various inputs at fixed points in the
parameter space in order to then drive the evolution through it. Populating
the vast space sufficiently densely is a very computationally expensive task,
and there are ongoing efforts to develop a cost-effective method to ease this
burden [66, 63, 64, 65, 67].

Meanwhile, there has been significant development of analytical techniques in
GSF theory, which invoke a ‘post-Newtonian’ (PN) framework of BH perturbation
theory [68, 69]. This PN-GSF approach is far less expensive than numerical GSF
calculations, readily covering a large region of parameter space that would be
difficult to populate numerically [70, 71, 72, 73, 74, 75]. To date, PN-GSF
results have informed EMRI waveform model indirectly through ‘effective one-
body’ [76, 77, 78, 79, 80] and ‘kludge’ models [81, 82, 83, 84, 85, 86, 87].
Despite being computationally cheap and much desired for LISA studies, the PN-
GSF framework has not yet been implemented for relativistic adiabatic
waveforms due to its technical complexity.

We here report the first adiabatic EMRI waveform model for generic Kerr orbits
based on the analytical PN-GSF approach. Our waveform is a standalone, on-the-
fly model within GSF theory, efficiently covering the weak-field region of
EMRI parameter space; see Figs. 1 and 2 below. Among other things, this allows
us to consistently account for the transient GSF resonances in the waveform
[30, 46, 88, 89, 90, 91]. These resonances will play a crucial role in the
detection and measurement of LISA EMRIs, but they have remained out of reach
in high-cost relativistic evolutions and have so far only been treated with
phenomenological models [92, 93, 94].

Furthermore, individual components of our model can be immediately combined
with fits to numerical GSF data and environmental effects [95, 96, 97, 98] to
construct both highly accurate and efficient waveform models [99, 100] that
will be employed in ongoing LISA preparatory studies (e.g., [101]), as well as
in eventual production-level code for LISA’s scientific analysis [102]. They
can be also used as reference points to inform the development of a universal
model of binaries across all mass ratios [103, 104, 105, 106]. We expect that
our PN-GSF approach will greatly improve the extensiveness and efficiency of
EMRI modelling for LISA, in much the same way that analytical relativistic
approaches have continued to advance modelling and data-analysis studies for
LIGO, Virgo, and KAGRA [107, 108, 109, 110].

Below we describe our adiabatic waveforms and assess their domain of validity.
Throughout we set $G=c=1$ and use $(t,\,r,\,\theta,\,\varphi)$ for Boyer-
Lindquist coordinates.

Snapshot waveforms from geodesic trajectories.—To set the stage, we first
summarize EMRI snapshot waveforms, where the small body’s motion is strictly
geodesic [54, 111, 57]. $M$ and $a$ denote the mass and spin parameters of the
Kerr BH; $q\equiv a/M$, its dimensionless spin magnitude; and $\mu\ll M$, the
mass of the small body (so $\mu/M=\eta$). Our convention is that $q>0$ ($q<0$)
represents prograde (retrograde) orbits.

A bound Kerr geodesic is confined to a toroidal region given by
$r_{\mathrm{min}}\leq r\leq r_{\mathrm{max}}$ and $\theta_{\rm
min}\leq\theta\leq\pi-\theta_{\rm min}$. The orbit is generically triperiodic,
uniquely described by three constants of motion
$I_{A}\equiv\\{p,\,e,\,\iota\\}$ [112, 43] and three orbital phases
$\Phi^{A}=\\{\Phi^{r},\Phi^{\theta},\Phi^{\varphi}\\}$ with constant
frequencies ${\rm d}\Phi^{A}/{\rm d}t=\Omega^{A}(I_{B})$ [113]. The three
constant orbital parameters are the semi-latus rectum
$Mp\equiv(2r_{\mathrm{max}}\,r_{\mathrm{min}})/(r_{\mathrm{max}}+r_{\mathrm{min}})$,
orbital eccentricity
$e\equiv({r_{\mathrm{max}}-r_{\mathrm{min}}})/({r_{\mathrm{max}}+r_{\mathrm{min}}})$,
and inclination angle $\tan\iota\equiv{\sqrt{\hat{C}}}/{{\hat{L}}}$, where
${\hat{L}}$ and ${\hat{C}}$ are the specific azimuthal angular momentum and
Carter constant of the geodesic [114]. The orbit’s radial, polar, and
azimuthal positions are $2\pi$-periodic in $\Phi^{r}$, $\Phi^{\theta}$, and
$\Phi^{\varphi}$, respectively.

The gravitational radiation from these geodesics can be conveniently computed
in the Teukolsky BH perturbation formalism, working with the linear
perturbation of the Weyl scalar $\psi_{4}=O(\eta)$ [115, 116]; at infinity,
the two GW polarizations $h_{+,\,\times}$ are simply given by
$\psi_{4}(r\to\infty)=\frac{1}{2}\frac{\partial^{2}}{\partial
t^{2}}({h}_{+}-i{h}_{\times})$. $\psi_{4}$ in the Fourier domain admits a full
separation of variables by means of spin-weighted ($s=-2$) spheroidal
harmonics ${{}_{-2}S_{\ell m}^{a\omega}}(\theta)$. The source geodesic’s
triperiodicity restricts the perturbation to the discrete frequency spectrum
$\omega_{mkn}\equiv m\,\Omega^{\varphi}+k\,\Omega^{\theta}+n\,\Omega^{r}$ for
integers $m,\,k,\,n$. We may thus write $h_{+,\,\times}$ as a multipolar sum
of “voices” with those frequencies: ${h}_{+}-i{h}_{\times}\propto
r^{-1}\sum_{{\ell mkn}}({{Z}_{{\ell mkn}}^{\infty}}{}_{-2}S_{\ell
m}^{a{\omega}_{mkn}}/{{\omega}^{2}_{mkn}})\,e^{-i\omega_{mkn}(t-r^{*})+im\varphi}$
with $\sum_{{\ell
mkn}}\equiv\sum_{\ell=2}^{\infty}\sum_{m=-\ell}^{\ell}\sum_{k=-\infty}^{\infty}\sum_{n=-\infty}^{\infty}$
and the Kerr tortoise coordinate $r^{*}$ [117]. Here, the dimensionless
asymptotic amplitudes at infinity ${Z}_{{\ell mkn}}^{\infty}(I_{A})$ are
obtained by solving the inhomogeneous radial Teukolsky equation mode by mode
with a fixed geodesic source, enforcing outgoing (ingoing) boundary conditions
at infinity (the horizon). The waveform phase $\omega_{mkn}\cdot(t-r^{*})$ is
a simple linear combination of the orbital phases, $\Phi_{mkn}(t)\equiv
m\,\Phi^{\varphi}(t)+k\,\Phi^{\theta}(t)+n\,\Phi^{r}(t)$, evaluated at the
retarded time $u=t-r^{*}$.

Adiabatic waveforms from inspiral trajectories.—We now turn to adiabatic
waveforms from orbits that slowly inspiral due to first-order [$O(\eta)$] GSF
effects [17, 19, 20, 21], specifically following the two-timescale framework
developed in Refs. [38, 39, 41]. In this framework, we introduce a ‘slow time’
${\tilde{t}}\equiv\eta t$. The trajectory and metric are treated as functions
of slowly varying parameters
${I}_{A}(\tilde{t})\equiv\\{p({\tilde{t}}),\,e({\tilde{t}}),\,\iota({\tilde{t}})\\}$
and ‘fast time’ phases $\Phi^{A}$ that evolve with slowly varying frequencies:

$\frac{{\rm d}\Phi^{A}}{{\rm d}t}=\Omega^{A}[{I}_{B}(\eta
t)]\quad\text{or}\quad\frac{{\rm d}\Phi^{A}}{{\rm
d}\tilde{t}}=\eta^{-1}\Omega^{A}[{I}_{B}(\tilde{t})]\,,$ (1)

where $\Omega^{A}({I}_{B})$ is the same function of $I_{B}$ as for a geodesic.
The dependence on $I_{A}$ captures the system’s evolution on the radiation-
reaction time scale $\sim M/\eta$, and the dependence on $\Phi^{A}$ capture
the triperiodicity on the orbital time scale $\sim M$.

Adiabatic inspirals and waveforms in this framework can be heuristically
understood as a slow-time evolution through the space of geodesic snapshots.
The self-forced equations of motion and Teukolsky equations are split into
slow- and fast-time equations. At each slow time step ${\tilde{t}}$, the
leading-order fast-time equations are identical to the equations for a
geodesic snapshot, yielding the same Teukolsky amplitudes as a snapshot with
the same parameters $I_{A}$. The adiabatic waveform can then be written in a
form precisely analogous to the snapshot waveforms described above [41, 118],

${h}_{+}-i{h}_{\times}=-\frac{2\,\mu}{r}\sum_{{\ell mkn}}\,\frac{{Z}_{{\ell
mkn}}^{\infty}}{{\omega}^{2}_{mkn}}\,\frac{{}_{-2}S_{\ell
m}^{a{\omega}_{mkn}}}{\sqrt{2\pi}}\,e^{-i{\Phi}_{mkn}+im\varphi}\,,$ (2)

where ${Z}_{{\ell mkn}}^{\infty}$, ${\omega}^{2}_{mkn}$, and ${}_{-2}S_{\ell
m}^{a{\omega}_{mkn}}$ are all geodesic functions of $I_{A}$, and the snapshot
phase $\omega_{mkn}\cdot(t-r^{*})$ is now replaced by the adiabatic phase
$\Phi_{mkn}({\tilde{u}})$ [satisfying Eq. (1)] evaluated at the slow retarded
time $\tilde{u}\equiv\eta\,(t-r^{*})$.

The evolution of $I_{A}$ can be derived from the self-forced equation of
motion for the Boyer-Lindquist coordinate trajectory $z^{\mu}$. If we define
osculating parameters $I^{\rm osc}_{A}$ by the condition that $z^{\mu}(I^{\rm
osc}_{A},\Phi^{A})$ and ${\rm d}z^{\mu}/{\rm d}t(I^{\rm osc}_{A},\Phi^{A})$
satisfy the geodesic relationships between $\\{z^{\mu},{\rm d}z^{\mu}/{\rm
d}t\\}$ and $\\{I_{A},\Phi^{A}\\}$, then we can straightforwardly derive an
equation of the form ${\rm d}I^{\rm osc}_{A}/{\rm d}t=G_{A}\sim\eta$ [41]. If
we write $I^{\rm osc}_{A}$ and $G_{A}$ as Fourier series of the form
$G_{A}=\sum_{{\mathsf{k}}_{r}{\mathsf{k}}_{\theta}}G_{A}^{{\mathsf{k}}_{r}{\mathsf{k}}_{\theta}}(I_{B})\,\exp{\\{i({\mathsf{k}}_{r}\,\Phi^{r}+{\mathsf{k}}_{\theta}\,\Phi^{\theta})\\}}$,
then at leading order the slowly varying $I_{A}$ is the stationary, $00$ mode
of $I^{\rm osc}_{A}$, and its driving force is the $00$ mode of $G_{A}$.
$G^{00}_{A}$ involves only the dissipative piece of $G_{A}$ [44, 46, 30, 47,
38] , which allows us to express it in terms of the asymptotic flux of
radiation [111] in the convenient ‘flux-balance’ form [119, 120, 121]:

$\frac{{\rm d}{I}_{A}}{{\rm d}{\tilde{t}}}=-M\,\sum_{{\ell
mkn}}\,\frac{({\beta}_{mkn})_{A}}{4\pi{\omega}^{3}_{mkn}}\left\\{|{Z}_{\ell
mkn}^{\infty}|^{2}+{\alpha}_{\ell mkn}\,|{Z}_{\ell
mkn}^{\mathrm{H}}|^{2}\right\\}\,,$ (3)

where ${Z}_{\ell mkn}^{\mathrm{H}}(I_{A})$ is the Teukolsky amplitude of
$\psi_{4}$ at the horizon, and ${\alpha}_{{\ell mkn}}$ and
$({\beta}_{mkn})_{A}$ are certain functions of ${I}_{A}$ [71]. The adiabatic
evolution is then given by Eqs. (1)–(3).

Transient resonances.—However, the adiabatic-evolution scheme described above
has to be altered if, at some instant ${\tilde{t}}={\tilde{t}}_{\rm res}$, the
slowly evolving orbital frequencies satisfy

$\omega^{\rm res}({\tilde{t}}_{\rm
res})\equiv{\beta^{r}}{\Omega^{r}}({\tilde{t}}_{\rm
res})-{\beta^{\theta}}{\Omega^{\theta}}({\tilde{t}}_{\rm res})=0\,,$ (4)

with a pair of nonzero coprime integers $(\beta^{r},\,\beta^{\theta})$. This
condition leads to a transient resonance of GSF effects that occurs for
generic EMRIs [30, 46, 88, 89, 90, 91]. At the resonance, otherwise-
oscillatory modes of $G_{A}$ with
${\mathsf{k}}_{r}:{\mathsf{k}}_{\theta}={\beta}^{r}:-\beta^{\theta}$ become
stationary, and the flux-balance formulas (3) are then enhanced (or
diminished) according to [122, 123, 124, 125, 126, 127, 121]

$\frac{{\rm d}{I}_{A}^{{\rm res}}}{{\rm d}t}\equiv\left.\sum_{s\neq
0}G_{A}^{{\rm res},s}\,e^{i\,s\,\Phi^{\rm
res}}\right|_{{\tilde{t}}={\tilde{t}}_{\rm res}}\,.$ (5)

Here, $G_{A}^{{\rm res},s}\equiv G^{s\beta_{r},-s\beta_{\theta}}_{A}$ is the
$s$th resonant mode of $G_{A}$, and the phase $\Phi^{\rm
res}\equiv\beta^{r}\,\Phi^{r}-\beta^{\theta}\,\Phi^{\theta}$ becomes
stationary at resonance.

Dissipation drives the orbit through the resonance, with a resonance-crossing
time [92, 93]

${\tilde{T}}^{{\rm res},\,s}\equiv\sqrt{\frac{2\pi}{|s\,{\dot{\omega}}^{{\rm
res}}|}}\sim M\,\eta^{-1/2}\,,$ (6)

where ${\dot{\omega}}^{{\rm res}}\equiv{{\rm d}{\omega}^{\rm res}/{{\rm
d}\tilde{t}}}$ evaluated at the resonance. $\tilde{T}^{{\rm res},\,s}$ is
longer than the orbital time scale $\sim M$, but much shorter than the
radiation-reaction time scale $\sim M/\eta$. On the radiation-reaction time,
the resonance crossing is effectively instantaneous, and the correction (5)
causes a sudden jump $\delta I_{A}^{\rm res}$ at ${\tilde{t}}={\tilde{t}}_{\rm
res}$. Formally expanding the phase $\Phi^{\rm res}(t)$ around the resonance
as $\Phi^{\rm res}(t_{\rm res})+\frac{1}{2}{\dot{\omega}}^{{\rm res}}(t-t_{\rm
res})^{2}+O(|t-t_{\rm res}|^{3})$ with the aid of Eqs. (1) and (4), and
integrating Eq. (5) under the stationary phase approximation, we find (see,
e.g., Refs. [34, 41])

$\delta I_{A}^{\rm res}=\sum_{s\neq 0}\left.\,{\tilde{T}}^{{\rm
res},\,s}\,{G_{A}^{{\rm res},s}}\,e^{{\rm sgn}\left(s\,{\dot{\omega}}^{{\rm
res}}\right)\frac{i\pi}{4}+i\,s\,\Phi^{\rm
res}}\,\right|_{{\tilde{t}}={\tilde{t}}_{\rm res}}\sim\eta^{1/2}\,.$ (7)

This induces corresponding abrupt frequency jumps
$\delta\Omega^{A}\sim\eta^{1/2}/M$, resulting in large cumulative phase shifts
$\delta\Phi^{A}\sim\eta^{-1/2}$ a radiation-reaction time after the resonance-
crossing, which will deteriorate our ability to measure EMRIs.

If we only have access to the leading-order dissipative GSF, we cannot
calculate the exact $O(\eta^{1/2})$ jumps for two reasons. First, the size of
$\delta I_{A}^{\rm res}$ in Eq. (7) sensitively depends on the phase
$\Phi^{\rm res}$. Calculating the jump therefore requires knowing the phase
through first post-adiabatic order in the evolution preceding the resonance
[128]. Second, the conservative GSF directly contributes to $G_{A}^{{\rm
res},s}$ [129, 121]. Accounting for these effects is beyond the current state
of the art.

Nevertheless, Eq. (7) correctly determines the jump that is internally
consistent with an adiabatic phase evolution. Similarly, discarding the
conservative contribution to Eq. (7) yields the correct jump associated with
the dissipative GSF. That dissipative piece of $G_{A}^{{\rm res},s}$ can be
constructed directly from the Teukolsky amplitudes as [130]

$\displaystyle G_{A}^{{\rm res},s}(I_{B})=-M\,\sum_{{\ell
mkn}}\,\sum_{\begin{subarray}{c}k^{\prime}=k-s\beta^{\theta}\\\
n^{\prime}=n+s\beta^{r}\end{subarray}}\frac{({b}_{mkn})_{A}+{k^{\prime}}\,({c}_{mkn})_{A}}{4\pi{\omega}^{3}_{mkn}}$
(8) $\displaystyle\times\left\\{{\rm Re}\left({Z}_{\ell
mkn}^{\infty}\,\overline{{Z}_{\ell
mk^{\prime}n^{\prime}}^{\infty}}\right)+{\alpha}_{\ell mkn}\,{\rm
Re}\left({Z}_{\ell mkn}^{\rm H}\,\overline{{Z}_{\ell
mk^{\prime}n^{\prime}}^{\rm H}}\right)\right\\}\,,$ (9)

where the overline denotes complex conjugation, and
$\\{({b}_{mkn})_{A},\,(c_{mkn})_{A}\\}$ are certain functions of ${I}_{A}$.
Because both conservative and dissipative contributions are comparable (at
least in a scalar-field toy model [131]), we expect the jumps calculated in
this way should qualitatively capture the impact of a resonance crossing, at
the order-of-magnitude level.

Techniques.—The end-to-end implementation of Eqs. (1), (2), (3), and (7) to
generate adiabatic waveforms needs three main inputs across the full parameter
space of Kerr spin $q$ and orbital parameters $I_{A}$: the frequencies
${\Omega}^{A}(I_{B})$, the spheroidal harmonics ${}_{-2}S_{\ell
m}^{a{\omega}_{mkn}}$, and the Teukolsky amplitudes ${Z}_{{\ell
mkn}}^{\infty,\,{\mathrm{H}}}$. The expression for ${\Omega}^{A}(I_{B})$ is
given in closed analytical form [113, 132]. We can also analytically compute
${}_{-2}S_{\ell m}^{a{\omega}_{mkn}}$ and ${Z}_{{\ell
mkn}}^{\infty,\,{\mathrm{H}}}$, building on the semi-analytical method of
solving the Teukolsky equation [133, 134, 135] in a small-frequency expansion.
This calculation is performed with the analytical module of the Black Hole
Perturbation Club (BHPC) code [36] developed in Refs. [136, 137, 138, 139,
140, 119, 141, 142, 143, 70, 71]. It assumes the ‘spheroidicity’ $a\omega$,
‘velocity’ ${v}\equiv\sqrt{1/{p}}$, and eccentricity ${e}$ are much smaller
than unity but allows arbitrary inclination ${\iota}$ and spin $|q|<1$ [141].
We obtain ${}_{-2}S_{\ell m}^{a{\omega}_{mkn}}$ up to $(a\omega_{mkn})^{4}$
(so $\sim v^{12}$), and ${Z}_{{\ell mkn}}^{\infty,\,{\mathrm{H}}}$ through
$v^{10}$ and $e^{10}$ beyond their leading order ‘Newtonian-circular’ terms
[144]. This ‘$5$PN-$e^{10}$’ calculation includes harmonic modes of
$2\leq\ell\leq 12$ with $|m|\leq\ell$, $|m+k|\leq 12$ and $|n|\leq 10$, which
gives $\approx 33,000$ nontrivial modes in total after exploiting mode
symmetries [54, 57, 26, 145].

With the analytical inputs in hand, we numerically evolve the system of GW
phases (1), polarizations (2), and orbital parameters (3) in the slow time
$\tilde{t}$; this numerical evolution (mostly) avoids the $5$PN-$e^{10}$
expansion of ${\Omega}^{A}(I_{B})$, which would severely limit waveform
accuracy. The evolution starts with given initial parameters $I_{A}(0)\equiv
I_{A}({\tilde{t}}=0)$ and phases $\Phi^{A}(0)\equiv\Phi^{A}(\tilde{t}=0)$ and
halts when it satisfies the resonance condition (4), at which time the jump
$\delta I_{A}^{\rm res}$ is calculated from Eq. (7). We then resume the
evolution from ${\tilde{t}}={\tilde{t}}_{\rm res}$ with the shifted orbital
parameters $I_{A}({\tilde{t}_{\rm res}})+\delta I_{A}^{\rm res}$. These
procedures are repeated until the evolution reaches a chosen termination time.
The nonresonant part of the evolution is implemented for CPUs, and it is
competitive with the numerical kludge code [84], which is the ‘fastest’ (semi-
relativistic) on-the-fly waveform so far.

Domain of validity.—Before presenting our adiabatic waveform, we compare
against an ‘exact’ numerical adiabatic data set to assess the accuracy of our
$5$PN-$e^{10}$ Teukolsky amplitudes and fluxes.

In doing so, we also employ the numerical module of the BHPC code [36, 53,
146, 57, 63] to compute ${}_{-2}S_{\ell m}^{a{\omega}_{mkn}}$ for viewing
angles $0^{\circ}\leq\theta\leq 90^{\circ}$, and ${Z}_{{\ell
mkn}}^{\infty,\,{\mathrm{H}}}$ for $q=\\{0,\,\pm 0.3,\,\pm 0.5,\,\pm 0.7,\,\pm
0.9\\}$ and
$\iota\sim\\{0^{\circ},\,20^{\circ},\,40^{\circ},\,60^{\circ},\,80^{\circ}\\}$
on a grid in $(p,\,e)$, where $6\lesssim p\lesssim 30$ and $0.01\lesssim
e\lesssim 0.4$. The harmonic content of the numerical data is dynamically
determined by the condition that the corresponding fluxes ${\cal F}_{A}^{\rm
num}\equiv{{\rm d}{I}_{A}}/{{\rm d}{\tilde{t}}}$ obtained from Eq. (3) have
fractional accuracy $\lesssim 10^{-5}$. We then use two figures of merit for
the $5$PN-$e^{10}$ analytical results: (i) the ‘mode-distribution error’ of
vectorized amplitudes $H\equiv{\rm{vec}}\left({{Z}_{{\ell
mkn}}^{\infty}}\,{}_{-2}S_{\ell
m}^{a{\omega}_{mkn}}/{{\omega}^{2}_{mkn}}\right)$ defined by $|1-\Re{(H^{{\rm
ana}},\,H^{\rm num})}/(|H^{\rm ana}|\,|H^{\rm num}|)|$, where the inner
product $(\cdot\,,\cdot)$ and its associated norm $|\cdot|$ are (implicitly)
Hermitian [64], and (ii) the relative flux error of $I_{A}$ defined by
$|1-{\cal F}_{A}^{\rm ana}/{\cal F}_{A}^{\rm num}|$.

Figures 1 and 2 show example results of the mode-distribution error and
relative flux error, respectively [147]. In addition to the obvious limitation
in the small-$p$ (hence large-$v$) and large-$e$ parameter regions, the errors
are also larger with decreasing $\iota$ or $q$ when the last stable geodesic
orbit lies at $p\gtrsim 6.0$ [148]; the location of this strong-field orbit is
difficult to capture with a small-$(v,\,e)$ expansion. In Fig. 2, we also test
Teukolsky-fitted fluxes used in kludge models [81, 83, 84]. In general, the
$5$PN-$e^{10}$ fluxes are more accurate than the fitted fluxes for $e\gtrsim
0.2$, independent of $q$ and $\iota\neq 0$.

Figure 1: Mode distribution error of $5$PN-$e^{10}$ amplitudes relative to
numerical data. We show the best case of $q=+0.9$, $\iota\approx 80^{\circ}$
(Left) and the worst case of $q=-0.9$, $\iota\approx 20^{\circ}$ (Right); for
the latter, the last stable orbit always lies at $p>8.5$. The dashed curves
indicate an error of $\approx 1.0\times 10^{-3}$, and at $p=20$ they cross
$e\approx 0.33$ ($e\approx 0.35$) in the left (right) panel. The plots are for
$(\theta,\,\varphi)=(45^{\circ},\,0^{\circ})$ but do not depend strongly on
the viewing angle because our expansion of ${}_{-2}S_{\ell
m}^{a{\omega}_{mkn}}$ accounts for higher $(a\omega_{mkn})^{4}$-corrections,
valid up to $7$PN.  Figure 2: Error of $5$PN-$e^{10}$ fluxes ${\cal
F}_{p}(\equiv{{\rm d}p}/{{\rm d}{\tilde{t}}})$ relative to numerical flux
data, with the same parameters as those in Fig. 1. The dashed curves indicate
an error of $\approx 1.0\times 10^{-2}$, while the dotted curves show, for
reference, the same level of error for Teukolsky-fitted fluxes [81, 83, 84].

For parameter inference of LISA-type EMRIs, a mode distribution error of
$\lesssim 1.0\times 10^{-3}$ is adequate [64, 67]. Inference requirements are
far more stringent for the fluxes, but these will not be attained even with
‘exact’ adiabatic models. Comparisons to numerical adiabatic evolutions of
equatorial orbits [63] suggest that relative flux errors $\lesssim 10^{-2}$
(of $dp/d{\tilde{t}}$) will suffice for our waveform to maintain phase
coherence with adiabatic LISA-EMRI waveforms over several months; this will
also be the level of agreement between adiabatic and post-adiabatic models
[149]. We therefore estimate the domain of validity of the $5$PN-$e^{10}$
results as $6\lesssim p\lesssim 20$ and $0\lesssim e\lesssim 0.3$ across
$|q|\leq 0.9$ and $0^{\circ}\leq\iota\leq 80^{\circ}$, excluding the parameter
region near the last stable orbit.

Sample results.—As a concrete example, we present a $5$PN-$e^{10}$ adiabatic
waveform with masses and Kerr spin
$(\mu,\,M,\,q)=(10M_{\odot},\,10^{6}M_{\odot},\,0.9)$, initial orbital
parameters $(p(0),\,e(0),\,\iota(0))=(9.6,\,0.21,\,80^{\circ})$, and initial
phases $\Phi^{A}(0)=0$. We evolve the EMRI over $\approx 4$ months, starting
from the initial apastron
$(r(0),\,\theta(0),\,\varphi(0))=[Mp(0)/(1-e(0)),\,\pi/2,\,0]$ with
$d\theta(0)/dt<0$, and ending with final orbital parameter values $(p_{\rm
f},\,e_{\rm f},\,\iota_{\rm f})\approx(9.23,\,0.197,\,80.1^{\circ})$. During
that time the inspiral passes through one strong, $3$:$2$ resonance; all other
resonances that it encounters are suppressed by additional powers of $v$ and
$e$ and contribute negligible jumps according to Eq. (7).

Figure 3 shows the first $\approx 11$ hours of the $5$PN-$e^{10}$ adiabatic
strain $h_{+}$. For reference, we also plot the snapshot strain from the fixed
geodesic source (i.e., without GSF effects) with the same initial frequencies,
generated by the BHPC’s numerical Teukolsky solver [57]; this serves as a
benchmark for the $5$PN-$e^{10}$ waveform so long as the viewing time is much
shorter than the dephasing time $\sim M/(\sqrt{\eta}\,\Omega^{A})$ [37, 34],
after which the inspiral orbit becomes $\sim 1$ cycle out of phase with the
geodesic orbit. The dephasing time of this sample EMRI is $\sim 2$ days. As
the figure shows, the $5$PN-$e^{10}$ model faithfully approximates the
numerical snapshot, a consequence of the small mode distribution error
$\approx 5.0\times 10^{-4}$ in the $5$PN-$e^{10}$ amplitudes.

Figure 3:  Left: $5$PN-$e^{10}$ adiabatic waveform of a sample EMRI with
masses $(M,\,\mu)=(10^{6}M_{\odot},\,10M_{\odot})$ and spin $q=0.9$, starting
at $(p_{0},\,e_{0},\,\iota_{0})=(9.6,\,0.21,\,80^{\circ})$. We plot the first
$\approx 11$ hours ($M\approx 5$ sec) of the evolving waveform at the viewing
angle $(\theta,\,\varphi)=(45^{\circ},\,0^{\circ})$. The rich structure of the
waveform is due to the beating of voice sets $\omega_{mkn}$, which encode, for
example, periastron precession $(\propto\Omega^{\varphi}-\Omega^{r})$ and
Lense-Thirring precession $(\propto\Omega^{\varphi}-\Omega^{\theta})$. The
dashed curve is the reference snapshot waveform from the fixed geodesic orbit
with the same initial frequencies and phases at $t=0$; this was generated
using the BHPC’s numerical Teukolsky code [57]. Right: the first $\approx 25$
minutes.

The slow evolution of the adiabatic waveform is more visible in the time-
frequency plot 4. The $3$:$2$ resonance occurs at ${\tilde{t}}_{\rm
res}\approx 4.3452\,M$, where there are abrupt frequency jumps
$(\delta\Omega^{r}/{\sqrt{\eta}},\,\delta\Omega^{\theta}/{\sqrt{\eta}},\,\delta\Omega^{\varphi}/{\sqrt{\eta}})\approx(4.54\times
10^{-4},\,1.02\times 10^{-3},\,1.13\times 10^{-3})$, corresponding to the
jumps $(\delta p^{\rm res}/{\sqrt{\eta}},\,\delta e^{\rm
res}/{\sqrt{\eta}},\,\delta{\iota}^{\rm res}/{\sqrt{\eta}})\approx(1.80\times
10^{-2},\,8.66\times 10^{-3},\,2.17^{\circ}\times 10^{-2})$ estimated from Eq.
(7). Although the frequency jumps are small $(\propto\sqrt{\eta})$, they lead
to large cumulative phase shifts $(\delta\Phi^{\rm res}_{r},\,\delta\Phi^{\rm
res}_{\theta},\,\delta\Phi^{\rm res}_{\varphi})\approx(2.38,\,5.49,\,6.13)$ by
the termination time ${\tilde{t}}=20.0\,M$. Such shifts are dramatic compared
to LISA’s EMRI phase resolution $\sim 0.1$ rad [150, 95, 96], reconfirming the
importance of GSF resonances for EMRI measurements [92, 93, 94].

Figure 4: Slowly evolving orbital frequencies
$\Omega^{A}\equiv\\{\Omega^{r},\,\Omega^{\theta},\,\Omega^{\varphi}\\}$ for
the sample EMRI waveform in Fig. 3 as a function of the slow time
${\tilde{t}}=\eta\,t$. The adiabatic evolution lasts for $\approx 4\text{
months}$ $({\tilde{t}}=20.0\,M)$. The vertical dashed line marks the $3$:$2$
resonance at ${\tilde{t}}_{\rm res}\approx 4.3452\,M$ with frequencies
$M\,\Omega^{r}\approx 2.11\times 10^{-2}$ and $M\,\Omega^{\theta}\approx
3.17\times 10^{-2}$. The inset enlarges the region near the resonance, showing
the difference $\delta\Omega^{A}:=M(\Omega^{A}_{\rm non-res}-\Omega^{A}_{\rm
res})$ between the evolution with and without resonance effects. At the
resonance, there are order-$\sqrt{\eta}$ discontinuous frequency jumps.

Concluding remarks.—Our PN-GSF adiabatic model represents the first user-
ready, relativistic description of EMRI waveforms in the astrophysical
scenario of generic Kerr orbits, including an approximate treatment of GSF
resonances. It can be used to generate on-the-fly waveforms over the whole
weak-field, small-eccentricity region of the EMRI parameter space, with
arbitrary orbital inclination and Kerr spin, thus opening a new front in
ongoing EMRI modelling and data analysis efforts [6, 101].

In the near term, we will improve our $5$PN-$e^{10}$ analytical calculations
to cover more of the EMRI parameter space [143, 70, 151, 152, 153, 154, 155].
We will further accelerate our model towards EMRI data-analysis, using the
efficiency-oriented FastEMRIWaveforms framework [100], which will enable a
highly parallelized implementation with graphics processing units [64, 67].
Ultimately, we will work on refining the adiabatic model by combining
analytical PN-GSF results with numerical GSF data [156, 157, 26, 158, 27, 28,
63, 65, 159, 28, 160, 161, 162], to accomplish a science-adequate, post-
adiabatic waveform for LISA.

Finally, it would be informative to compare our adiabatic evolution with
small-mass-ratio results from PN theory [163, 164] and fully nonlinear
numerical-relativity simulations [165, 166, 167, 168]. This may further
delineate the applicable region of GSF theory [169, 40] for generic binary
BHs.

###### Acknowledgements.

Acknowledgments.—We thank Wataru Hikida and Hideyuki Tagoshi for their direct
contributions to an earlier version of this manuscript; Scott A. Hughes for
helpful discussions on initial phases and comments on the Supplemental
Material; Maarten van de Meent for providing independent numerical data to
verify BHPC’s Teukolsky results; Leor Barack and Chulmoon Yoo for valuable
discussions and comments on the manuscript; and Katsuhiko Ganz, Chris
Kavanagh, Koutarou Kyutoku, Yasushi Mino, Takashi Nakamura, Misao Sasaki,
Masaru Shibata, and Niels Warburton for very helpful discussions. S.I. is
especially grateful to Eric Poisson, Riccardo Sturani and Takahiro Tanaka for
their continuous encouragement and insightful discussion about the (adiabatic)
evolution scheme for EMRI dynamics. Finally, we thank all the past and present
members of the annual Capra meetings with whom we have discussed the
techniques and results presented here (over the past decades). S.I.
acknowledges support from STFC through grant no. ST/R00045X/1, the GWverse
COST Action CA16104,“Black holes, gravitational waves and fundamental physic”,
and additional financial support of Ministry of Education - MEC during his
stay at IIP-Natal-Brazil. A.J.K.C. acknowledges support from the NASA grants
no. 18-LPS18-0027 and 20-LPS20-0005, and from the NSF grant no. PHY-2011968.
A.P. acknowledges the support of a Royal Society University Research
Fellowship, Research Grant for Research Fellows, Enhancement Awards, and
Exchange Grant. This work was supported in part by JSPS/MEXT KAKENHI Grant no.
JP16H02183 (R.F.), JP18H04583 (R.F.), JP21H01082 (R.F., H.N. and N.S.),
JP17H06358 (H.N. and N.S.), and JP21K03582 (H.N.).

## References

  * Abbott  _et al._ [2019] B. P. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. X 9, 031040 (2019), arXiv:1811.12907 [astro-ph.HE] .
  * Abbott  _et al._ [2021a] R. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. X 11, 021053 (2021a), arXiv:2010.14527 [gr-qc] .
  * Abbott  _et al._ [2021b] R. Abbott _et al._ , arXiv e-prints , arXiv:2108.01045 (2021b), arXiv:2108.01045 [gr-qc] .
  * Abbott  _et al._ [2021c] R. Abbott _et al._ , arXiv e-prints , arXiv:2111.03606 (2021c), arXiv:2111.03606 [gr-qc] .
  * Amaro-Seoane  _et al._ [2017] P. Amaro-Seoane _et al._ , arXiv e-prints , arXiv:1702.00786 (2017), arXiv:1702.00786 [astro-ph.IM] .
  * Kawamura  _et al._ [2021] S. Kawamura _et al._ , PTEP 2021, 05A105 (2021), arXiv:2006.13545 [gr-qc] .
  * Mei  _et al._ [2021] J. Mei _et al._ (TianQin), PTEP 2021, 05A107 (2021), arXiv:2008.10332 [gr-qc] .
  * Luo  _et al._ [2021] Z. Luo, Y. Wang, Y. Wu, W. Hu, and G. Jin, PTEP 2021, 05A108 (2021).
  * Gong  _et al._ [2021] Y. Gong, J. Luo, and B. Wang, Nature Astron. 5, 881 (2021), arXiv:2109.07442 [astro-ph.IM] .
  * Amaro-Seoane [2018] P. Amaro-Seoane, Living Rev. Rel. 21, 4 (2018), arXiv:1205.5240 [astro-ph.CO] .
  * Amaro-Seoane [2020] P. Amaro-Seoane, arXiv e-prints , arXiv:2011.03059 (2020), arXiv:2011.03059 [gr-qc] .
  * Amaro-Seoane  _et al._ [2007] P. Amaro-Seoane, J. R. Gair, M. Freitag, M. Coleman Miller, I. Mandel, C. J. Cutler, and S. Babak, Class. Quant. Grav. 24, R113 (2007), arXiv:astro-ph/0703495 .
  * Babak  _et al._ [2017] S. Babak, J. Gair, A. Sesana, E. Barausse, C. F. Sopuerta, C. P. L. Berry, E. Berti, P. Amaro-Seoane, A. Petiteau, and A. Klein, Phys. Rev. D 95, 103012 (2017), arXiv:1703.09722 [gr-qc] .
  * Berry  _et al._ [2019] C. Berry, S. Hughes, C. Sopuerta, A. Chua, A. Heffernan, K. Holley-Bockelmann, D. Mihaylov, C. Miller, and A. Sesana, Bull. Am. Astron. Soc. 51, 42 (2019), arXiv:1903.03686 [astro-ph.HE] .
  * Fan  _et al._ [2020] H.-M. Fan, Y.-M. Hu, E. Barausse, A. Sesana, J.-d. Zhang, X. Zhang, T.-G. Zi, and J. Mei, Phys. Rev. D 102, 063016 (2020), arXiv:2005.08212 [astro-ph.HE] .
  * Zi  _et al._ [2021] T.-G. Zi, J.-D. Zhang, H.-M. Fan, X.-T. Zhang, Y.-M. Hu, C. Shi, and J. Mei, Phys. Rev. D 104, 064008 (2021), arXiv:2104.06047 [gr-qc] .
  * Mino  _et al._ [1997a] Y. Mino, M. Sasaki, and T. Tanaka, Phys. Rev. D 55, 3457 (1997a), arXiv:gr-qc/9606018 .
  * Mino  _et al._ [1997b] Y. Mino, M. Sasaki, and T. Tanaka, Prog. Theor. Phys. Suppl. 128, 373 (1997b), arXiv:gr-qc/9712056 .
  * Quinn and Wald [1997] T. C. Quinn and R. M. Wald, Phys. Rev. D 56, 3381 (1997), arXiv:gr-qc/9610053 .
  * Gralla and Wald [2008] S. E. Gralla and R. M. Wald, Class. Quant. Grav. 25, 205009 (2008), [Erratum: Class.Quant.Grav. 28, 159501 (2011)], arXiv:0806.3293 [gr-qc] .
  * Pound [2010] A. Pound, Phys. Rev. D 81, 024023 (2010), arXiv:0907.5197 [gr-qc] .
  * Pound [2012] A. Pound, Phys. Rev. Lett. 109, 051101 (2012), arXiv:1201.5089 [gr-qc] .
  * Gralla [2012] S. E. Gralla, Phys. Rev. D 85, 124011 (2012), arXiv:1203.3189 [gr-qc] .
  * Shah  _et al._ [2012] A. G. Shah, J. L. Friedman, and T. S. Keidl, Phys. Rev. D 86, 084059 (2012), arXiv:1207.5595 [gr-qc] .
  * van de Meent [2016] M. van de Meent, Phys. Rev. D 94, 044034 (2016), arXiv:1606.06297 [gr-qc] .
  * van de Meent [2018] M. van de Meent, Phys. Rev. D 97, 104033 (2018), arXiv:1711.09607 [gr-qc] .
  * Pound  _et al._ [2020] A. Pound, B. Wardell, N. Warburton, and J. Miller, Phys. Rev. Lett. 124, 021101 (2020), arXiv:1908.07419 [gr-qc] .
  * Warburton  _et al._ [2021] N. Warburton, A. Pound, B. Wardell, J. Miller, and L. Durkan, Phys. Rev. Lett. 127, 151102 (2021), arXiv:2107.01298 [gr-qc] .
  * Mino [2005a] Y. Mino, Class. Quant. Grav. 22, S717 (2005a), arXiv:gr-qc/0506002 .
  * Tanaka [2006] T. Tanaka, Prog. Theor. Phys. Suppl. 163, 120 (2006), arXiv:gr-qc/0508114 .
  * Barack [2009] L. Barack, Class. Quant. Grav. 26, 213001 (2009), arXiv:0908.1664 [gr-qc] .
  * Poisson  _et al._ [2011] E. Poisson, A. Pound, and I. Vega, Living Rev. Rel. 14, 7 (2011), arXiv:1102.0529 [gr-qc] .
  * Barack  _et al._ [2019] L. Barack _et al._ , Class. Quant. Grav. 36, 143001 (2019), arXiv:1806.05195 [gr-qc] .
  * Barack and Pound [2019] L. Barack and A. Pound, Rept. Prog. Phys. 82, 016904 (2019), arXiv:1805.10385 [gr-qc] .
  * [35] Black Hole Perturbation Toolkit, http://bhptoolkit.org/.
  * Black Hole Perturbation Club () [B.H.P.C.] Black Hole Perturbation Club (B.H.P.C.), https://sites.google.com/view/bhpc1996/home.
  * Detweiler [2005] S. L. Detweiler, Class. Quant. Grav. 22, S681 (2005), arXiv:gr-qc/0501004 .
  * Hinderer and Flanagan [2008] T. Hinderer and E. E. Flanagan, Phys. Rev. D 78, 064028 (2008), arXiv:0805.3337 [gr-qc] .
  * Miller and Pound [2021] J. Miller and A. Pound, Phys. Rev. D 103, 064048 (2021), arXiv:2006.11263 [gr-qc] .
  * van de Meent and Pfeiffer [2020] M. van de Meent and H. P. Pfeiffer, Phys. Rev. Lett. 125, 181101 (2020), arXiv:2006.12036 [gr-qc] .
  * Pound and Wardell [2021] A. Pound and B. Wardell, arXiv e-prints , arXiv:2101.04592 (2021), arXiv:2101.04592 [gr-qc] .
  * Wardell  _et al._ [2021] B. Wardell, A. Pound, N. Warburton, J. Miller, L. Durkan, and A. Le Tiec, arXiv e-prints , arXiv:2112.12265 (2021), arXiv:2112.12265 [gr-qc] .
  * Hughes [2000] S. A. Hughes, Phys. Rev. D 61, 084004 (2000), [Erratum: Phys. Rev. D 63, 049902 (2001), Erratum: Phys. Rev. D 65, 069902 (2002), Erratum: Phys. Rev. D 67, 089901 (2003), Erratum: Phys. Rev. D 78, 109902 (2008), Erratum: Phys. Rev. D 90, 109904 (2014)], arXiv:gr-qc/9910091 .
  * Mino [2003] Y. Mino, Phys. Rev. D 67, 084027 (2003), arXiv:gr-qc/0302075 .
  * Hughes  _et al._ [2005] S. A. Hughes, S. Drasco, E. E. Flanagan, and J. Franklin, Phys. Rev. Lett. 94, 221101 (2005), arXiv:gr-qc/0504015 .
  * Mino [2005b] Y. Mino, Prog. Theor. Phys. 113, 733 (2005b), arXiv:gr-qc/0506003 .
  * Mino [2006] Y. Mino, Prog. Theor. Phys. 115, 43 (2006), arXiv:gr-qc/0601019 .
  * Mino [2008] Y. Mino, Phys. Rev. D 77, 044008 (2008), arXiv:0711.3007 [gr-qc] .
  * Nakamura  _et al._ [1987] T. Nakamura, K. Oohara, and Y. Kojima, Prog. Theor. Phys. Suppl. 90, 1 (1987).
  * Finn and Thorne [2000] L. S. Finn and K. S. Thorne, Phys. Rev. D 62, 124021 (2000), arXiv:gr-qc/0007074 .
  * Glampedakis and Kennefick [2002] K. Glampedakis and D. Kennefick, Phys. Rev. D 66, 044002 (2002), arXiv:gr-qc/0203086 .
  * Hughes [2001] S. A. Hughes, Phys. Rev. D 64, 064004 (2001), [Erratum: Phys.Rev.D 88, 109902 (2013)], arXiv:gr-qc/0104041 .
  * Fujita and Tagoshi [2004] R. Fujita and H. Tagoshi, Prog. Theor. Phys. 112, 415 (2004), arXiv:gr-qc/0410018 .
  * Drasco and Hughes [2006] S. Drasco and S. A. Hughes, Phys. Rev. D 73, 024027 (2006), [Erratum: Phys. Rev. D 88, 109905 (2013), Erratum: Phys. Rev. D 90, 109905 (2014)], arXiv:gr-qc/0509101 .
  * Sundararajan  _et al._ [2007] P. A. Sundararajan, G. Khanna, and S. A. Hughes, Phys. Rev. D 76, 104005 (2007), arXiv:gr-qc/0703028 .
  * Sundararajan  _et al._ [2008] P. A. Sundararajan, G. Khanna, S. A. Hughes, and S. Drasco, Phys. Rev. D 78, 024022 (2008), arXiv:0803.0317 [gr-qc] .
  * Fujita  _et al._ [2009] R. Fujita, W. Hikida, and H. Tagoshi, Prog. Theor. Phys. 121, 843 (2009), arXiv:0904.3810 [gr-qc] .
  * Harms  _et al._ [2013] E. Harms, S. Bernuzzi, and B. Brügmann, Class. Quant. Grav. 30, 115013 (2013), arXiv:1301.1591 [gr-qc] .
  * Harms  _et al._ [2014] E. Harms, S. Bernuzzi, A. Nagar, and A. Zenginoglu, Class. Quant. Grav. 31, 245004 (2014), arXiv:1406.5983 [gr-qc] .
  * Gralla  _et al._ [2016] S. E. Gralla, S. A. Hughes, and N. Warburton, Class. Quant. Grav. 33, 155002 (2016), [Erratum: Class.Quant.Grav. 37, 109501 (2020)], arXiv:1603.01221 [gr-qc] .
  * Burke  _et al._ [2020] O. Burke, J. R. Gair, and J. Simón, Phys. Rev. D 101, 064026 (2020), arXiv:1909.12846 [gr-qc] .
  * Gourgoulhon  _et al._ [2019] E. Gourgoulhon, A. Le Tiec, F. H. Vincent, and N. Warburton, Astron. Astrophys. 627, A92 (2019), arXiv:1903.02049 [gr-qc] .
  * Fujita and Shibata [2020] R. Fujita and M. Shibata, Phys. Rev. D 102, 064005 (2020), arXiv:2008.13554 [gr-qc] .
  * Chua  _et al._ [2021] A. J. K. Chua, M. L. Katz, N. Warburton, and S. A. Hughes, Phys. Rev. Lett. 126, 051102 (2021), arXiv:2008.06071 [gr-qc] .
  * Hughes  _et al._ [2021] S. A. Hughes, N. Warburton, G. Khanna, A. J. K. Chua, and M. L. Katz, Phys. Rev. D 103, 104014 (2021), arXiv:2102.02713 [gr-qc] .
  * Barton  _et al._ [2008] J. L. Barton, D. J. Lazar, D. J. Kennefick, G. Khanna, and L. M. Burko, Phys. Rev. D 78, 064042 (2008), arXiv:0804.1075 [gr-qc] .
  * Katz  _et al._ [2021] M. L. Katz, A. J. K. Chua, L. Speri, N. Warburton, and S. A. Hughes, Phys. Rev. D 104, 064047 (2021), arXiv:2104.04582 [gr-qc] .
  * Mino  _et al._ [1997c] Y. Mino, M. Sasaki, M. Shibata, H. Tagoshi, and T. Tanaka, Prog. Theor. Phys. Suppl. 128, 1 (1997c), arXiv:gr-qc/9712057 .
  * Sasaki and Tagoshi [2003] M. Sasaki and H. Tagoshi, Living Rev. Rel. 6, 6 (2003), arXiv:gr-qc/0306120 .
  * Fujita [2015] R. Fujita, PTEP 2015, 033E01 (2015), arXiv:1412.5689 [gr-qc] .
  * Sago and Fujita [2015] N. Sago and R. Fujita, PTEP 2015, 073E03 (2015), arXiv:1505.01600 [gr-qc] .
  * Kavanagh  _et al._ [2016] C. Kavanagh, A. C. Ottewill, and B. Wardell, Phys. Rev. D 93, 124038 (2016), arXiv:1601.03394 [gr-qc] .
  * Bini and Geralico [2018] D. Bini and A. Geralico, Found. Phys. 48, 1349 (2018).
  * Bini and Geralico [2019] D. Bini and A. Geralico, Phys. Rev. D 100, 104002 (2019), arXiv:1907.11080 [gr-qc] .
  * Munna [2020a] C. Munna,  __,Ph.D. thesis, North Carolina U. (2020a).
  * Yunes  _et al._ [2010] N. Yunes, A. Buonanno, S. A. Hughes, M. Coleman Miller, and Y. Pan, Phys. Rev. Lett. 104, 091102 (2010), arXiv:0909.4263 [gr-qc] .
  * Yunes  _et al._ [2011] N. Yunes, A. Buonanno, S. A. Hughes, Y. Pan, E. Barausse, M. Miller, and W. Throwe, Phys. Rev. D 83, 044044 (2011), [Erratum: Phys. Rev. D 88, 109904 (2013)], arXiv:1009.6013 [gr-qc] .
  * Xin  _et al._ [2019] S. Xin, W.-B. Han, and S.-C. Yang, Phys. Rev. D 100, 084055 (2019), arXiv:1812.04185 [gr-qc] .
  * Zhang  _et al._ [2021] C. Zhang, W.-B. Han, X.-Y. Zhong, and G. Wang, Phys. Rev. D 104, 024050 (2021), arXiv:2102.05391 [gr-qc] .
  * Albanesi  _et al._ [2021] S. Albanesi, A. Nagar, and S. Bernuzzi, Phys. Rev. D 104, 024067 (2021), arXiv:2104.10559 [gr-qc] .
  * Glampedakis  _et al._ [2002] K. Glampedakis, S. A. Hughes, and D. Kennefick, Phys. Rev. D 66, 064005 (2002), arXiv:gr-qc/0205033 .
  * Barack and Cutler [2004] L. Barack and C. Cutler, Phys. Rev. D 69, 082005 (2004), arXiv:gr-qc/0310125 .
  * Gair and Glampedakis [2006] J. R. Gair and K. Glampedakis, Phys. Rev. D 73, 064037 (2006), arXiv:gr-qc/0510129 .
  * Babak  _et al._ [2007] S. Babak, H. Fang, J. R. Gair, K. Glampedakis, and S. A. Hughes, Phys. Rev. D 75, 024005 (2007), [Erratum: Phys. Rev. D 77, 04990 (2008)], arXiv:gr-qc/0607007 .
  * Sopuerta and Yunes [2011] C. F. Sopuerta and N. Yunes, Phys. Rev. D 84, 124060 (2011), arXiv:1109.0572 [gr-qc] .
  * Chua and Gair [2015] A. J. Chua and J. R. Gair, Class. Quant. Grav. 32, 232002 (2015), arXiv:1510.06245 [gr-qc] .
  * Chua  _et al._ [2017] A. J. Chua, C. J. Moore, and J. R. Gair, Phys. Rev. D 96, 044005 (2017), arXiv:1705.04259 [gr-qc] .
  * Apostolatos  _et al._ [2009] T. A. Apostolatos, G. Lukes-Gerakopoulos, and G. Contopoulos, Phys. Rev. Lett. 103, 111101 (2009), arXiv:0906.0093 [gr-qc] .
  * Flanagan and Hinderer [2012] E. E. Flanagan and T. Hinderer, Phys. Rev. Lett. 109, 071102 (2012), arXiv:1009.4923 [gr-qc] .
  * Brink  _et al._ [2015a] J. Brink, M. Geyer, and T. Hinderer, Phys. Rev. Lett. 114, 081102 (2015a), arXiv:1304.0330 [gr-qc] .
  * Brink  _et al._ [2015b] J. Brink, M. Geyer, and T. Hinderer, Phys. Rev. D 91, 083001 (2015b), arXiv:1501.07728 [gr-qc] .
  * Ruangsri and Hughes [2014] U. Ruangsri and S. A. Hughes, Phys. Rev. D 89, 084036 (2014), arXiv:1307.6483 [gr-qc] .
  * Berry  _et al._ [2016] C. P. L. Berry, R. H. Cole, P. Cañizares, and J. R. Gair, Phys. Rev. D 94, 124042 (2016), arXiv:1608.08951 [gr-qc] .
  * Speri and Gair [2021] L. Speri and J. R. Gair, Phys. Rev. D 103, 124032 (2021), arXiv:2103.06306 [gr-qc] .
  * Bonga  _et al._ [2019] B. Bonga, H. Yang, and S. A. Hughes, Phys. Rev. Lett. 123, 101103 (2019), arXiv:1905.00030 [gr-qc] .
  * Gupta  _et al._ [2021] P. Gupta, B. Bonga, A. J. K. Chua, and T. Tanaka, Phys. Rev. D 104, 044056 (2021), arXiv:2104.03422 [gr-qc] .
  * Coogan  _et al._ [2022] A. Coogan, G. Bertone, D. Gaggero, B. J. Kavanagh, and D. A. Nichols, Phys. Rev. D 105, 043009 (2022), arXiv:2108.04154 [gr-qc] .
  * Barsanti  _et al._ [2022] S. Barsanti, N. Franchini, L. Gualtieri, A. Maselli, and T. P. Sotiriou, arXiv e-prints , arXiv:2203.05003 (2022), arXiv:2203.05003 [gr-qc] .
  * [99] EMRI Kludge Suite, https://github.com/alvincjk/EMRI_Kludge_Suite/.
  * [100] few: Fast EMRI Waveforms, https://bhptoolkit.org/FastEMRIWaveforms/html/index.html.
  * Chua and Cutler [2021] A. J. K. Chua and C. J. Cutler, arXiv e-prints , arXiv:2109.14254 (2021), arXiv:2109.14254 [gr-qc] .
  * [102] LISA Data Challenges, https://lisa-ldc.lal.in2p3.fr/.
  * Pan  _et al._ [2011] Y. Pan, A. Buonanno, R. Fujita, E. Racine, and H. Tagoshi, Phys. Rev. D 83, 064003 (2011), [Erratum: Phys. Rev. D 87, 109901 (2013)], arXiv:1006.0431 [gr-qc] .
  * Pratten  _et al._ [2020] G. Pratten, S. Husa, C. Garcia-Quiros, M. Colleoni, A. Ramos-Buades, H. Estelles, and R. Jaume, Phys. Rev. D 102, 064001 (2020), arXiv:2001.11412 [gr-qc] .
  * Rifat  _et al._ [2020] N. E. M. Rifat, S. E. Field, G. Khanna, and V. Varma, Phys. Rev. D 101, 081502 (2020), arXiv:1910.10473 [gr-qc] .
  * Islam  _et al._ [2022] T. Islam, S. E. Field, S. A. Hughes, G. Khanna, V. Varma, M. Giesler, M. A. Scheel, L. E. Kidder, and H. P. Pfeiffer, arXiv e-prints , arXiv:2204.01972 (2022), arXiv:2204.01972 [gr-qc] .
  * Ajith  _et al._ [2008] P. Ajith _et al._ , Phys. Rev. D 77, 104017 (2008), [Erratum: Phys.Rev.D 79, 129901 (2009)], arXiv:0710.2335 [gr-qc] .
  * Aylott  _et al._ [2009] B. Aylott _et al._ , Class. Quant. Grav. 26, 165008 (2009), arXiv:0901.4399 [gr-qc] .
  * Ajith  _et al._ [2012] P. Ajith _et al._ , Class. Quant. Grav. 29, 124001 (2012), [Erratum: Class. Quant. Grav. 30, 199401 (2013)], arXiv:1201.5319 [gr-qc] .
  * Hinder  _et al._ [2014] I. Hinder _et al._ , Class. Quant. Grav. 31, 025012 (2014), arXiv:1307.5307 [gr-qc] .
  * Sago  _et al._ [2005] N. Sago, T. Tanaka, W. Hikida, and H. Nakano, Prog. Theor. Phys. 114, 509 (2005), arXiv:gr-qc/0506092 .
  * Cutler  _et al._ [1994] C. Cutler, D. Kennefick, and E. Poisson, Phys. Rev. D 50, 3816 (1994).
  * Schmidt [2002] W. Schmidt, Class. Quant. Grav. 19, 2743 (2002), arXiv:gr-qc/0202090 .
  * Carter [1968] B. Carter, Phys. Rev. 174, 1559 (1968).
  * Teukolsky [1972] S. Teukolsky, Phys. Rev. Lett. 29, 1114 (1972).
  * Teukolsky [1973] S. A. Teukolsky, Astrophys. J. 185, 635 (1973).
  * Teukolsky and Press [1974] S. Teukolsky and W. Press, Astrophys. J. 193, 443 (1974).
  * See Supplemental Material at [2021a] See Supplemental Material at, [URL will be inserted by publisher] (2021a), for a comparison with another adiabatic-evolution scheme in Ref. [65].
  * Sago  _et al._ [2006] N. Sago, T. Tanaka, W. Hikida, K. Ganz, and H. Nakano, Prog. Theor. Phys. 115, 873 (2006), arXiv:gr-qc/0511151 .
  * Drasco  _et al._ [2005] S. Drasco, E. E. Flanagan, and S. A. Hughes, Class. Quant. Grav. 22, S801 (2005), arXiv:gr-qc/0505075 .
  * Isoyama  _et al._ [2019] S. Isoyama, R. Fujita, H. Nakano, N. Sago, and T. Tanaka, PTEP 2019, 013E01 (2019), arXiv:1809.11118 [gr-qc] .
  * Gair  _et al._ [2012] J. Gair, N. Yunes, and C. M. Bender, J. Math. Phys. 53, 032503 (2012), arXiv:1111.3605 [gr-qc] .
  * Grossman  _et al._ [2013] R. Grossman, J. Levin, and G. Perez-Giz, Phys. Rev. D 88, 023002 (2013), arXiv:1108.1819 [gr-qc] .
  * Flanagan  _et al._ [2014] E. E. Flanagan, S. A. Hughes, and U. Ruangsri, Phys. Rev. D 89, 084028 (2014), arXiv:1208.3906 [gr-qc] .
  * Isoyama  _et al._ [2013] S. Isoyama, R. Fujita, H. Nakano, N. Sago, and T. Tanaka, PTEP 2013, 063E01 (2013), arXiv:1302.4035 [gr-qc] .
  * van de Meent [2014] M. van de Meent, Phys. Rev. D 89, 084033 (2014), arXiv:1311.4457 [gr-qc] .
  * Mihaylov and Gair [2017] D. P. Mihaylov and J. R. Gair, J. Math. Phys. 58, 112501 (2017), arXiv:1706.06639 [gr-qc] .
  * Lukes-Gerakopoulos and Witzany [2020] G. Lukes-Gerakopoulos and V. Witzany, Nonlinear effects in emri dynamics and their imprints on gravitational waves, in  _Handbook of Gravitational Wave Astronomy_, edited by C. Bambi, S. Katsanevas, and K. D. Kokkotas (Springer Singapore, Singapore, 2020) pp. 1–44.
  * Fujita  _et al._ [2017] R. Fujita, S. Isoyama, A. Le Tiec, H. Nakano, N. Sago, and T. Tanaka, Class. Quant. Grav. 34, 134001 (2017), arXiv:1612.02504 [gr-qc] .
  * See Supplemental Material at [2021b] See Supplemental Material at, [URL will be inserted by publisher] (2021b), for a sketch of the derivation of Eq. (8).
  * Nasipak and Evans [2021] Z. Nasipak and C. R. Evans, Phys. Rev. D 104, 084011 (2021), arXiv:2105.15188 [gr-qc] .
  * Fujita and Hikida [2009] R. Fujita and W. Hikida, Class. Quant. Grav. 26, 135002 (2009), arXiv:0906.1420 [gr-qc] .
  * Fackerell and Crossman [1977] E. D. Fackerell and R. G. Crossman, Journal of Mathematical Physics 18, 1849 (1977).
  * Mano  _et al._ [1996] S. Mano, H. Suzuki, and E. Takasugi, Prog. Theor. Phys. 95, 1079 (1996), arXiv:gr-qc/9603020 .
  * Mano and Takasugi [1997] S. Mano and E. Takasugi, Prog. Theor. Phys. 97, 213 (1997), arXiv:gr-qc/9611014 .
  * Shibata  _et al._ [1995] M. Shibata, M. Sasaki, H. Tagoshi, and T. Tanaka, Phys. Rev. D 51, 1646 (1995), arXiv:gr-qc/9409054 .
  * Tagoshi [1995] H. Tagoshi, Prog. Theor. Phys. 93, 307 (1995), [Erratum: Prog. Theor. Phys. 118, 577–579 (2007)].
  * Tagoshi  _et al._ [1996] H. Tagoshi, M. Shibata, T. Tanaka, and M. Sasaki, Phys. Rev. D 54, 1439 (1996), arXiv:gr-qc/9603028 .
  * Tanaka  _et al._ [1996] T. Tanaka, H. Tagoshi, and M. Sasaki, Prog. Theor. Phys. 96, 1087 (1996), arXiv:gr-qc/9701050 .
  * Tagoshi  _et al._ [1997] H. Tagoshi, S. Mano, and E. Takasugi, Prog. Theor. Phys. 98, 829 (1997), arXiv:gr-qc/9711072 .
  * Ganz  _et al._ [2007] K. Ganz, W. Hikida, H. Nakano, N. Sago, and T. Tanaka, Prog. Theor. Phys. 117, 1041 (2007), arXiv:gr-qc/0702054 .
  * Fujita [2012a] R. Fujita, Prog. Theor. Phys. 127, 583 (2012a), arXiv:1104.5615 [gr-qc] .
  * Fujita [2012b] R. Fujita, Prog. Theor. Phys. 128, 971 (2012b), arXiv:1211.5535 [gr-qc] .
  * The $5$PN-$e^{10}$ results for the spheroidal harmonics ${}_{-2}S_{\ell m}^{a{\omega}}$, Teukolsky amplitudes ${Z}_{{\ell mkn}}^{\infty}$, and fluxes ${{\cal F}_{A}}$ used in this work are all available for download at [2021] The $5$PN-$e^{10}$ results for the spheroidal harmonics ${}_{-2}S_{\ell m}^{a{\omega}}$, Teukolsky amplitudes ${Z}_{{\ell mkn}}^{\infty}$, and fluxes ${{\cal F}_{A}}$ used in this work are all available for download at, [https://sites.google.com/view/bhpc1996/data] (2021).
  * Nasipak  _et al._ [2019] Z. Nasipak, T. Osburn, and C. R. Evans, Phys. Rev. D 100, 064008 (2019), arXiv:1905.13237 [gr-qc] .
  * Fujita and Tagoshi [2005] R. Fujita and H. Tagoshi, Prog. Theor. Phys. 113, 1165 (2005), arXiv:0904.3818 [gr-qc] .
  * See Supplemental Material at [2021c] See Supplemental Material at, [URL will be inserted by publisher] (2021c), for additional plots of relative flux errors.
  * Stein and Warburton [2020] L. C. Stein and N. Warburton, Phys. Rev. D 101, 064007 (2020), arXiv:1912.07609 [gr-qc] .
  * See Supplemental Material at [2021d] See Supplemental Material at, [URL will be inserted by publisher] (2021d), which includes Refs. [170, 171, 172, 173], for an estimate of the dephasing between adiabatic and first post-adiabatic model.
  * Lindblom  _et al._ [2008] L. Lindblom, B. J. Owen, and D. A. Brown, Phys. Rev. D 78, 124020 (2008), arXiv:0809.3844 [gr-qc] .
  * Shah [2014] A. G. Shah, Phys. Rev. D 90, 044025 (2014), arXiv:1403.2697 [gr-qc] .
  * Munna and Evans [2019] C. Munna and C. R. Evans, Phys. Rev. D 100, 104060 (2019), arXiv:1909.05877 [gr-qc] .
  * Munna  _et al._ [2020] C. Munna, C. R. Evans, S. Hopper, and E. Forseth, Phys. Rev. D 102, 024047 (2020), arXiv:2005.03044 [gr-qc] .
  * Munna [2020b] C. Munna, Phys. Rev. D 102, 124001 (2020b), arXiv:2008.10622 [gr-qc] .
  * Munna and Evans [2020] C. Munna and C. R. Evans, Phys. Rev. D 102, 104006 (2020), arXiv:2009.01254 [gr-qc] .
  * Warburton  _et al._ [2012] N. Warburton, S. Akcay, L. Barack, J. R. Gair, and N. Sago, Phys. Rev. D 85, 061501 (2012), arXiv:1111.6908 [gr-qc] .
  * Osburn  _et al._ [2016] T. Osburn, N. Warburton, and C. R. Evans, Phys. Rev. D 93, 064024 (2016), arXiv:1511.01498 [gr-qc] .
  * Van De Meent and Warburton [2018] M. Van De Meent and N. Warburton, Class. Quant. Grav. 35, 144003 (2018), arXiv:1802.05281 [gr-qc] .
  * McCart  _et al._ [2021] J. McCart, T. Osburn, and J. Y. J. Burton, Phys. Rev. D 104, 084050 (2021), arXiv:2109.00056 [gr-qc] .
  * Lynch  _et al._ [2021] P. Lynch, M. van de Meent, and N. Warburton, arXiv e-prints , arXiv:2112.05651 (2021), arXiv:2112.05651 [gr-qc] .
  * Mathews  _et al._ [2022] J. Mathews, A. Pound, and B. Wardell, Phys. Rev. D 105, 084031 (2022), arXiv:2112.13069 [gr-qc] .
  * Skoupý and Lukes-Gerakopoulos [2022] V. Skoupý and G. Lukes-Gerakopoulos, Phys. Rev. D 105, 084033 (2022), arXiv:2201.07044 [gr-qc] .
  * Will and Maitra [2017] C. M. Will and M. Maitra, Phys. Rev. D 95, 064003 (2017), arXiv:1611.06931 [gr-qc] .
  * Tucker and Will [2021] A. Tucker and C. M. Will, Phys. Rev. D 104, 104023 (2021), arXiv:2108.12210 [gr-qc] .
  * Lewis  _et al._ [2017] A. G. M. Lewis, A. Zimmerman, and H. P. Pfeiffer, Class. Quant. Grav. 34, 124001 (2017), arXiv:1611.03418 [gr-qc] .
  * Fernando  _et al._ [2019] M. Fernando, D. Neilsen, H. Lim, E. Hirschmann, and H. Sundar, SIAM J. Sci. Comput. 41, C97 (2019), arXiv:1807.06128 [gr-qc] .
  * Lousto and Healy [2020] C. O. Lousto and J. Healy, Phys. Rev. Lett. 125, 191102 (2020), arXiv:2006.04818 [gr-qc] .
  * Lousto and Healy [2022] C. O. Lousto and J. Healy, arXiv e-prints , arXiv:2203.08831 (2022), arXiv:2203.08831 [gr-qc] .
  * Le Tiec [2014] A. Le Tiec, Int. J. Mod. Phys. D 23, 1430022 (2014), arXiv:1408.5505 [gr-qc] .
  * Barack and Sago [2009] L. Barack and N. Sago, Phys. Rev. Lett. 102, 191101 (2009), arXiv:0902.0573 [gr-qc] .
  * Le Tiec  _et al._ [2012] A. Le Tiec, E. Barausse, and A. Buonanno, Phys. Rev. Lett. 108, 131103 (2012), arXiv:1111.5609 [gr-qc] .
  * Isoyama  _et al._ [2014] S. Isoyama, L. Barack, S. R. Dolan, A. Le Tiec, H. Nakano, A. G. Shah, T. Tanaka, and N. Warburton, Phys. Rev. Lett. 113, 161101 (2014), arXiv:1404.6133 [gr-qc] .
  * van de Meent [2017] M. van de Meent, Phys. Rev. Lett. 118, 011101 (2017), arXiv:1610.03497 [gr-qc] .


11affiliationtext: Interdisciplinary Program in Artificial Intelligence
Seoul National University 22affiliationtext: NAVER WEBTOON AI
33affiliationtext: LG Electronics

# MDPose: Real-Time Multi-Person Pose Estimation
via Mixture Density Model

Seunghyeon Seo Jaeyoung Yoo Jihye Hwang Nojun Kwak

###### Abstract

One of the major challenges in multi-person pose estimation is instance-aware
keypoint estimation. Previous methods address this problem by leveraging an
off-the-shelf detector, heuristic post-grouping process or explicit instance
identification process, hindering further improvements in the inference speed
which is an important factor for practical applications. From the statistical
point of view, those additional processes for identifying instances are
necessary to bypass learning the high-dimensional joint distribution of human
keypoints, which is a critical factor for another major challenge, the
occlusion scenario. In this work, we propose a novel framework of single-stage
instance-aware pose estimation by modeling the joint distribution of human
keypoints with a mixture density model, termed as MDPose. Our MDPose estimates
the distribution of human keypoints’ coordinates using a mixture density model
with an instance-aware keypoint head consisting simply of 8 convolutional
layers. It is trained by minimizing the negative log-likelihood of the ground
truth keypoints. Also, we propose a simple yet effective training strategy,
Random Keypoint Grouping (RKG), which significantly alleviates the underflow
problem leading to successful learning of relations between keypoints. On
OCHuman dataset, which consists of images with highly occluded people, our
MDPose achieves state-of-the-art performance by successfully learning the
high-dimensional joint distribution of human keypoints. Furthermore, our
MDPose shows significant improvement in inference speed with a competitive
accuracy on MS COCO, a widely-used human keypoint dataset, thanks to the
proposed much simpler single-stage pipeline.

##  1 Introduction

Multi-person pose estimation is a classical computer vision task that aims to
localize human keypoints in an image. As it is a fundamental computer vision
problem leading to various practical applications such as action recognition,
human-computer interaction and so on, it has been studied actively since the
development of deep learning.

One of the major challenges in multi-person pose estimation is instance-aware
keypoint estimation and many works have been studied to tackle this problem,
which can be categorized into two major paradigms: top-down [xiao2018simple,
SunXLW19, li2021human, papandreou2017towards, chen2018cascaded,
khirodkar2021multi] and bottom-up approaches [Varamesh_2020_CVPR,
zhou2019centernet, cao2017realtime, Kreiss_2019_CVPR, cheng2020higherhrnet,
GengSXZW21, newell2017associative]. As shown in Fig. 1 (a) and (b), the top-
down method exploits an off-the-shelf detector and the bottom-up method
performs a post-grouping process for a common goal of instance specification.
However, there exist some bottlenecks toward the efficient instance-aware
keypoint estimation. Since the top-down method is a two-stage method which
detects a person then localizes its keypoints one by one, the more the number
of people in an image, the slower the inference speed. In the case of the
bottom-up method, it depends on a post-grouping process, which is usually
heuristic and takes additional time for keypoint refinement for the instance-
aware keypoint estimation.

Figure 1: Illustration of multi-person pose estimation frameworks: (a) Top-
down, (b) Bottom-up, (c) previous Single-stage Instance-aware and (d) Ours.
The colored boxes indicate the process for identifying instances, which we
successfully removed by proposing a mixture-model-based architecture.

Recently, there have been approaches to tackle the aforementioned weaknesses
for instance-aware keypoint estimation [tian2019directpose, mao2021fcpose], as
shown in Fig. 1 (c). mao2021fcpose proposed FCPose, a single-stage instance-
aware framework based on FCOS detector [tian2019fcos], equipped with a dynamic
keypoint head consisting of instance-specific weights. Since it leverages the
capacity of FCOS detector and is a one-stage method at the same time, it can
achieve a reasonably high accuracy at a relatively fast inference speed.
However, it still relies on the detector’s performance for generating instance
weights and such instance identification process hinders further improvement
in the inference speed.

In this paper, we propose a novel multi-person pose estimation framework using
a mixture model. There has been a line of research utilizing the mixture model
in various pose estimation tasks [Li_2019_CVPR, deepdirectstat2018,
Ye_2018_ECCV, Varamesh_2020_CVPR]. Among them, MDN3 [Varamesh_2020_CVPR]
showed the potential in the multi-person pose estimation task by modeling the
mixture model with a person’s viewpoint as a dominant factor. However, it lags
behind other state-of-the-art methods in terms of accuracy and inference
speed.

Figure 2: The overall architecture of MDPose. The parameters of mixture model
($\mu$, $\gamma$, $o$ and $\pi$) are obtained from a keypoint head consisting
of 8 convolutional layers. The mixture components are located along the
spatial axis, i.e. the number of mixture components in a feature map is
$H^{f}\times W^{f}$.

Inspired by Yoo_2021_ICCV, which showed competitive performance with a
mixture-model-based architecture in object detection, we propose a simple
architecture modeling joint distribution of human keypoints with a mixture
model, coined as MDPose. From the statistical point of view, previous methods
need to implement an additional instance identification process to bypass
learning high-dimensional joint distribution of human keypoints’ coordinates,
since the numerical underflow problem usually occurs during the training
process due to the curse of dimensionality. However, unless the high-
dimensional distribution is considered sufficiently, the performance
degradation is inevitable under the condition of severe occlusion. To tackle
this problem, we propose Random Keypoint Grouping (RKG) which learns the joint
distribution of continuously changing subsets of keypoints at every iteration.
It alleviates the underflow problem efficiently and leads to the successful
learning of relations between keypoints in the high-dimensional space, which
increases the capacity for distinguishing multiple occluded persons.
Furthermore, since a mixture component corresponds to a person, we can perform
instance-aware keypoint estimation without any additional instance
identification process, as shown in Fig. 1 (d). As a result, we could achieve
competitive performances with a simple instance-aware keypoint head consisting
of only 8-convolutional layer enabling real-time applications. Additionally,
unlike SunXLW19, xiao2018simple, cheng2020higherhrnet, cao2017realtime,
he2017mask, GengSXZW21, newell2017associative, MDPose does not need likelihood
heatmap during training which requires burdensome computational cost and
storage. In short, MDPose shows strong potential for practical applications
regarding both training and inference time as well as an occlusion condition.

Our MDPose performs instance-aware keypoint estimation without bells and
whistles through a mixture model framework. Our RKG makes it possible to learn
high-dimensional joint distribution of human keypoints’ coordinates,
eliminating additional instance identification processes. Specifically, on the
OCHuman [Zhang_2019_CVPR] validation and test set consisting of images with
heavily occluded persons, our MDPose achieves state-of-the-art performance
with 43.5 mAPkp and 42.7 mAPkp, respectively, by successfully learning human
keypoint representation in a high-dimensional space. Furthermore, on the COCO
keypoint validation set [lin2014mscoco], our MDPose achieves 64.6 mAPkp at the
speed of 29.8 FPS with a ResNet-50 backbone [he2016deep], which outperforms
other state-of-the-art methods by a large margin in inference speed.

##  2 Related works

#### Multi-person pose estimation.

One of the major challenges in multi-person pose estimation is to correctly
estimate keypoints per each person, i.e. instance-aware keypoint estimation.
Many studies have been done to address this problem which can be classified
into two paradigms: top-down and bottom-up approaches. The top-down approach
[papandreou2017towards, chen2018cascaded, xiao2018simple, SunXLW19,
li2021human, khirodkar2021multi] leverages an off-the-shelf detector to
localize an instance and performs a single-person pose estimation. While it
can achieve high accuracy, its inference speed is much slower than bottom-up
approaches, especially for an image with a large number of people. On the
other hand, the bottom-up approach [newell2017associative, zhou2019centernet,
cao2017realtime, Kreiss_2019_CVPR, cheng2020higherhrnet, GengSXZW21,
xue2022learning] performs instance-agnostic keypoint estimation and assigns
them to each instance through a post-grouping process. It shows more robust
and faster inference speed than top-down approaches. However, the post-
grouping process is usually heuristic and complicated with many
hyperparameters.

#### Single-stage instance-aware pose estimation.

Recently, there have been single-stage instance-aware approaches to tackle the
aforementioned drawbacks of existing frameworks [tian2019directpose,
mao2021fcpose]. Among them, mao2021fcpose proposed end-to-end trainable FCPose
which performs instance-aware keypoint estimation by a dynamic keypoint head
consisting of instance-specific weights generated by FCOS detector
[tian2019fcos]. As a result, it achieves competitive accuracy and inference
speed while eliminating heuristic post-grouping process. However, it still
depends on the performance of the object detector and the instance-specific
weight generation process remains as a bottleneck for further improvement of
inference speed.

#### Occluded pose estimation.

There are various approaches [jin2020differentiable, khirodkar2021multi,
li2019crowdpose, qiu2020peeking, Zhang_2019_CVPR] to improve performance in
occluded human pose estimation, which is another major challenge.
jin2020differentiable proposed a hierarchical graph grouping method to learn
relationship between keypoints in the bottom-up style. Among the top-down
methods, khirodkar2021multi introduced a Multi-Instance Modulation Block which
adjusts feature responses to distinguish multiple instances in a given
bounding box. Although they improve performance in the occlusion condition by
specifically devised methods or architectures, they still lack enough
consideration for learning the high-dimensional distribution of keypoints,
which is a fundamental challenge in the multi-person pose estimation.

##  3 Method

In this work, we propose a novel framework for learning the joint distribution
of human keypoints using a mixture model, leading to eliminating explicit
instance identification processes and boosting the capacity of distinguishing
occluded persons. Our MDPose is modeled with a mixture distribution so that
the mixture component corresponds to a person, i.e. one-to-one matching
between mixture components and persons, resulting in instance-aware keypoint
estimation without bells and whistles. Since it depends on neither an off-the-
shelf detector nor a post-grouping process, it can achieve a much simpler
pipeline with an accelerated speed than previous methods.

First, we will describe the mixture model and our problem formulation in Sec.
3.1 and propose a new architecture and describe it in detail in Sec. 3.2.
After that, we will explain the Random Keypoint Grouping (RKG) strategy for
learning the high-dimensional joint distribution and our final loss function
in Sec. 3.3. Finally, an inference phase will be described in Sec. 3.4.

###  3.1 Mixture model

In an image $X$, there exists a ground truth for each of $N$ persons,
$k^{gt}=\\{k^{gt}_{1},\cdots,k^{gt}_{N}\\}$, and $i$-th ground truth
$k^{gt}_{i}$ contains the keypoint coordinates
$k_{i}^{gt}=\\{k_{i,1,x},k_{i,1,y},\cdots,k_{i,K,x},k_{i,K,y}\\}$, where $K$
denotes the number of keypoints. Our MDPose estimates the distribution of
keypoint locations $k_{i}$ on an image $X$ with a mixture model. Based on the
design of the mixture model for object detection in Yoo_2021_ICCV, we develop
the architecture for the multi-person pose estimation task. Our mixture model
is formed by a weighted combination of component distributions, which we set
as a Laplace distribution. Although the Laplace distribution has a similar
shape with the Gaussian and the Cauchy distribution, its tails fall off more
rapidly than the Cauchy but less sharply than the Gaussian. We empirically
found that the Laplace distribution is more suitable for the multi-person pose
estimation task than the Gaussian and Cauchy. Related experimental results are
provided in the supplementary material. Every element of $k_{i}$ is assumed to
be independent111Although each element of a mixture component is independent
of others, they are jointly dependent in the overall joint distribution. of
each other to keep the mixture model from being over-complicated. Therefore,
the probability density function (pdf) of Laplace distribution is defined as,

$\displaystyle\mathcal{F}(k_{i};\mu,\gamma)$
$\displaystyle=\prod_{j=1}^{K}\prod_{d\in
D}\mathcal{F}(k_{i,j,d};\mu_{j,d},\gamma_{j,d})$ (1)
$\displaystyle=\prod_{j=1}^{K}\prod_{d\in
D}\frac{1}{2\gamma_{j,d}}\exp\left(-\frac{|k_{i,j,d}-\mu_{j,d}|}{\gamma_{j,d}}\right)$

with a set of keypoint coordinates $D=\\{x,y\\}$, where $j$ and $\mathcal{F}$
are the keypoint index and the Laplacian pdf, respectively. As a result, the
2$K$-dimensional Laplace represents the distribution of human keypoints
coordinates and the pdf of our mixture model is as follows:

$\displaystyle p(k^{gt}_{i}|X)$
$\displaystyle=\sum_{m=1}^{M}\pi_{m}\mathcal{F}(k_{i};\mu_{m},\gamma_{m}),$
(2)

where the $m$ denotes the index of $M$ mixture components.

###  3.2 Architecture

Fig. 2 demonstrates the overall architecture of our MDPose. The feature maps
are forwarded into the keypoint head to obtain intermediate outputs:
$\mu\textquoteright$, $\gamma\textquoteright$, and $o\textquoteright$. The
final outputs $\mu$, $\gamma$, $o$, and $\pi$ are obtained from intermediate
outputs as parameters of our mixture model. The mixture components are
represented at each position of the cells on the feature map, i.e. located
along the spatial axis.

The mean $\mu$ is derived from $\mu\textquoteright\in\mathbb{R}^{H^{f}\times
W^{f}\times 2K}$, where $H^{f}$ and $W^{f}$ indicate the height and width of a
feature map in the feature pyramid, respectively, and note that the number of
mixture components in a feature map is $H^{f}\times W^{f}$. Based on the
implementation of Yoo_2021_ICCV, $\mu\textquoteright$ is scaled by a factor of
$s=2^{l-5}$, where $l\in\\{1,\cdots,5\\}$ denotes the level of feature map in
the feature pyramid. Then, the scaled $\mu\textquoteright$ is added to
$\bar{\mu}\in\mathbb{R}^{H^{f}\times W^{f}\times 2K}$ which is the default
coordinates uniformly distributed in a grid pattern over the entire feature
map. In short, the final location parameter $\mu$ is obtained as follows:
$\mu=\bar{\mu}+s\mu^{\prime}$. We can obtain the positive scale parameter
$\gamma\in\mathbb{R}^{H^{f}\times W^{f}\times 2K}$ through softplus
[dugas2000softplus] activation function from $\gamma\textquoteright$. The
foreground probability $o\in\mathbb{R}^{H^{f}\times W^{f}\times 1}$ is
calculated by applying the sigmoid function to $o\textquoteright$. Following
yoo2022sparsemdod, we use the normalized foreground probability as $\pi$:
$\pi_{m}=o_{m}/\sum_{n=1}^{M}o_{n}$. Since the mixture components in a
foreground area are likely to have higher $\pi$, we can consider $\pi$ as the
normalized foreground probability so that $\sum_{m}^{M}\pi_{m}=1$.

The keypoint head of MDPose consists of eight 3x3 kernel convolutional layers
with Swish [ramachandran2017swish] activation function except the last layer.
The 5-level Feature Pyramid Network [lin2017fpn] is used as our feature
extractor. Since we estimate a mixture distribution from all-level feature
maps, the total number of mixture components is equal to the summation of the
number of mixture components in each level of feature map:
$M=\sum_{l=1}^{5}(H^{f}_{l}\times W^{f}_{l})$.

Figure 3: Illustration of RKG at an iteration with $K_{g}=3$ and $N_{g}=6$.
$k_{i,j}$ is a human keypoint, where $i$ and $j$ denote a person in an image
and a keypoint index, respectively. For the simplicity of grouping, we set the
center coordinate of the bounding box as $k_{i,18}$.

###  3.3 Training

Our MDPose is trained to maximize the likelihood of $k^{gt}$ for an input
image $X$. Therefore, we can simply define the loss function for minimizing
the negative log-likelihood (NLL) of $k^{gt}$ as follows:

$\displaystyle\mathcal{L}_{NLL}$ $\displaystyle=-\log
p(k^{gt}|X)=-\log\prod_{i=1}^{N}p(k^{gt}_{i}|X)$ (3)
$\displaystyle=-\sum_{i=1}^{N}\log\sum_{m=1}^{M}\pi_{m}\mathcal{F}(k_{i};\mu_{m},\gamma_{m}).$

Although the foreground probability $o$ is not used, it is trained through the
mixture coefficient $\pi$, i.e. the probability of a mixture component
[yoo2022sparsemdod].

In the training using (3), the curse of dimensionality arises due to the high-
dimensional joint distribution of human keypoints, e.g. 34 dimension in the
case of 17 keypoints in COCO keypoint dataset [lin2014mscoco], leading to a
severe underflow problem. As a result, it is extremely hard to compute
$\mathcal{L}_{NLL}$ via a 2$K$-dimensional joint distribution in the multi-
person pose estimation task.

#### Random keypoint grouping (RKG).

To tackle this problem, we propose RKG. As illustrated in Fig. 3, we shuffle
and split $K$ keypoints into $N_{g}$ groups, each consisting of $K_{g}$
keypoints, where $N_{g}$ and $K_{g}$ denote the number of groups and the
number of keypoints in a group, respectively, i.e. $K_{g}\times N_{g}=K$. As a
result, we can notate a set of keypoints’ indices in a group $g$ as $I_{g}$
and reformulate (1) using a group of keypoints as follows:

$\displaystyle\mathcal{F}(k^{g}_{i};\mu^{g},\gamma^{g})=\prod_{j\in
I_{g}}\prod_{d\in D}\mathcal{F}(k_{i,j,d};\mu_{j,d},\gamma_{j,d}),$ (4)

where the superscript $g$ indicates the index of the group. Therefore, we can
alleviate the underflow problem with 2$K_{g}$-dimensional joint distribution,
whose dimension is lower than the original 2$K$ dimension if $K_{g}<K$. Our
final loss function with RKG is defined as follows:

$\displaystyle\mathcal{L}_{GroupNLL}$
$\displaystyle=\frac{\sum_{g=1}^{N_{g}}\mathcal{L}_{NLL}^{g}}{N_{g}}$ (5)
$\displaystyle=-\frac{1}{N_{g}}\sum_{i=1}^{N}\sum_{g=1}^{N_{g}}\log\sum_{m=1}^{M}\pi_{m}\mathcal{F}(k^{g}_{i};\mu^{g}_{m},\gamma^{g}_{m}).$

Note that RKG is used only for the training process and the combination of
keypoints for a group changes at every iteration. As shown in (5), the RKG
amounts to factorizing the original joint distribution of 2$K$ dimension into
$N_{g}$ marginal distributions of 2$K_{g}$ dimension. Although each keypoints
group is estimated independently at each iteration, the keypoints end up being
dependent on each other through the whole training process due to RKG, which
keeps shuffling and grouping keypoints randomly. As a result, MDPose is able
to learn the relations between keypoints without any heuristic grouping
process. To ease the grouping scheme for COCO keypoint dataset [lin2014mscoco]
labeled with 17 keypoints, we use the coordinates of bounding box center of
$k^{gt}_{i}$ as an auxiliary keypoint only for training, which is denoted as
$k_{i,18}$ in Fig. 3.

###  3.4 Inference

In the inference phase, a mixture component of our MDPose corresponds to an
instance, i.e. a person in the multi-person pose estimation task. Therefore,
MDPose is able to perform an instance-aware keypoint estimation without bells
and whistles. $\mu$ and $o$ are used as the estimated keypoint coordinates and
confidence scores, respectively. Note that we do not use $\mu$ of the bbox
center coordinates for inference. Our final predictions are obtained by
removing duplicate estimations using non-maximum suppression (NMS), which is
applied to pseudo-bboxes, each of which consists of the minimum and the
maximum coordinates among keypoints as the left-top and the bottom-right
coordinates, respectively.

##  4 Experiments

###  4.1 Experimental details

#### Dataset.

We evaluate MDPose on the widely-used human keypoint dataset, MS COCO
[lin2014mscoco], consisting of 200K images including 250K person instances
labeled with 17 keypoints. Following the standard protocol, we split the
dataset into 57K images for training, 5K images for validation, and 20K images
for test-dev set. We adopt the average precision (AP) based on the object
keypoint similarity (OKS) as our evaluation metric. We conduct the analysis
for our MDPose on the validation set and compare with other state-of-the-art
methods on the test-dev set. Furthermore, we evaluate MDPose on OCHuman
[Zhang_2019_CVPR], which is a testing-only dataset focusing on the heavy
occlusion scenarios. It consists of 4,731 images with 8,110 person instances
labeled with 17 keypoints like MS COCO. While less than 1% of person instances
have occlusions with maxIoU $\geq$ 0.5 in MS COCO, all instances have
occlusions with maxIoU $\geq$ 0.5 and 32% of them are more challenging with
maxIoU $\geq$ 0.75 in OCHuman. Following Zhang_2019_CVPR, we use only MS COCO
train set for training and evaluate on OCHuman validation and test set.

#### Training.

As mentioned in 3.1, we represent the distribution of keypoint coordinates as
a Laplace distribution. We set $K_{g}=3$ and $N_{g}=6$ for RKG as our default
setting. We conduct experiments with different backbones including
ResNet-50/101 [he2016deep] and DLA-34 [Yu_2018_CVPR], which is especially for
further improvement of inference speed. All backbones are pretrained with
ImageNet [deng2009imagenet] and FPN [lin2017fpn] is used as the feature
extractor. For data augmentation, we apply random rotation in [-30∘, 30∘],
expand, random crop in [0.3, 1.0] (relative range) and random flip. Unless
specified, the input image is resized to 320$\times$320 for the analysis of
the RKG and mixture distributions or 896$\times$896 for the analysis of
inference speed and occluded pose estimation and comparison with other
methods. Following Yoo_2021_ICCV, MDPose is trained by SGD with a weight decay
of 5e-5 and gradient clipping with an L2 norm of 7.0. The batch size is 32 and
the synchronized batch normalization [Peng_2018_CVPR] is used for a consistent
learning behavior over different numbers of GPUs. The initial learning rate is
set to 0.01 which is reduced by a factor of 10 at the 180K and 240K iteration
in the training schedule of total 270K iterations.

#### Inference.

For inference, we use the same size of an image as in the training phase. The
mixture components with low confidence scores in $o$ are filtered out and NMS
is applied for removing duplicate estimations. We set thresholds of $o$ and
NMS as 1e-4 and 0.7, respectively. Note that our model does not have any
explicit process for identifying instance, such as post-grouping, weight
generation and so on.

Table 1: The performance according to the number of keypoints per group. $K_{g}$ and $N_{g}$ denote the number of keypoints per group and the total number of groups, respectively. $K_{g}$ | $N_{g}$ |  APkp |  AP${}_{50}^{kp}$ |  AP${}_{75}^{kp}$ |  AP${}_{M}^{kp}$ |  AP${}_{L}^{kp}$
---|---|---|---|---|---|---
1 | 18 | 47.7 | 76.7 | 50.2 | 37.6 | 61.7
2 | 9 | 51.2 | 79.6 | 54.1 | 41.3 | 64.9
3 | 6 | 51.5 | 80.4 | 55.1 | 42.0 | 64.7
6 | 3 | 51.0 | 80.1 | 54.5 | 41.2 | 64.3
9 | 2 | 49.8 | 78.9 | 53.4 | 40.5 | 62.8
18 | 1 | NaN | NaN | NaN | NaN | NaN
Table 2: Randomness of grouping strategy. Non-random indicates the heuristic grouping method which predefines the keypoints per group based on the relations of human body joints. Randomness |  APkp |  AP${}_{50}^{kp}$ |  AP${}_{75}^{kp}$ |  AP${}_{M}^{kp}$ |  AP${}_{L}^{kp}$
---|---|---|---|---|---
Non-random | 39.5 | 69.9 | 40.0 | 32.4 | 49.7
Random | 51.5 | 80.4 | 55.1 | 42.0 | 64.7

###  4.2 Analysis of RKG

#### The number of keypoints per group.

We conducted an analysis for the number of keypoints per group, $K_{g}$. Since
the number of groups, $N_{g}$, is determined according to $K_{g}$, i.e.
$K_{g}\times N_{g}=K$, the more the number of keypoints in a group, the higher
the joint distribution’s dimension is.

In Tab. 1, it shows the best performance of 51.5 APkp with RKG of $K_{g}=3$,
which we set as our default setting. Fig. 4 shows the trade-off between the
accuracy and the ratio of numerically underflowed components. When we apply
RKG of $K_{g}=1$ or $2$, the performance is inferior to that of $K_{g}=3$
despite the lower underflow ratio since our MDPose with RKG of high $K_{g}$
can learn the relations of keypoints more efficiently by modeling the joint
distribution with more keypoints. In particular, although there is no
underflow problem due to the low dimension of joint distribution with RKG of
$K_{g}=1$, it cannot learn the relations of keypoints sufficiently during the
training, leading to notably lower APkp than RKG of $K_{g}=2$ and $3$ as shown
in Tab. 1.

However, with RKG of more than $K_{g}=3$, our MDPose suffers from the
underflow problem as $K_{g}$ increases, and the performance is rather lower
than that with RKG of $K_{g}=3$. As expected, with RKG of $K_{g}=18$, i.e.
with only one group, the original joint distribution is impossible to learn,
resulting in NaN in Tab. 1. It is due to the severe underflow problem caused
by the curse of dimensionality, i.e. the underflow ratio is 1.0 as shown in
Fig. 4.

Figure 4: The trade-off between accuracy and ratio of underflowed components.
Table 3: Inference speed comparison with other methods on COCO val set.

Method Backbone APkp FPS CenterNet [zhou2019centernet] Hourglass 64.0 6.8 DEKR
[GengSXZW21] HRNet-W32 68.0 8.1 HRNet-W48 71.0 5.2 FCPose [mao2021fcpose]
ResNet-50 63.0 20.7 SimpleBaseline [xiao2018simple] ResNet-50 72.4 6.8
ResNet-101 73.4 5.3 ResNet-152 74.3 4.0 PifPaf [Kreiss_2019_CVPR] ResNet-152
67.4 4.7 MDPose (Ours) ResNet-50 64.6 29.8 ResNet-101 65.2 20.8 DLA-34 64.2
58.9

#### Randomness in the grouping.

Tab. 2 compares RKG with non-random grouping, which forms a group
heuristically based on the relations of human body joints, i.e. $N_{g}=6$
groups of left arm, left leg, right arm, right leg, eyes and nose, and ears
and bbox center, each consisting of $K_{g}=3$ keypoints. In comparison to the
MDPose with non-random grouping, RKG improves the performance significantly
from 39.5 APkp to 51.5 APkp. While non-random grouping learns only the joint
distributions of pre-defined adjacent keypoint groups, RKG enables learning of
the overall joint distributions of every non-adjacent keypoints through the
whole training process by randomly grouping at every iteration. The comparison
through qualitative results is provided in the supplementary material.

Figure 5: FPS by the number of people in an image. Figure 6: Comparison
between FCPose and MDPose in the occlusion scenario. FCPose and MDPose are
shown on the 1st and 2nd row, respectively. The red circles in (b) show the
differences of the estimated results for occluded keypoints between FCPose and
MDPose.

###  4.3 Analysis of the inference speed

Tab. 3 presents the comparison with other methods on COCO val set. The FPS is
measured on a single NVIDIA TITAN RTX. Ours achieves 64.6 APkp and 29.8 FPS
with ResNet-50 backbone, which is comparable or superior to others especially
in inference speed. It is 44%-faster than FCPose with an identical backbone,
which is a single-stage instance-aware method enabling real-time application.
Furthermore, we implement MDPose with DLA-34 as a backbone to further boost
the inference speed. Following tian2019fcos, we adopt the 3-level FPN and a
training schedule of 360K iterations with learning rate decay by a factor of
10 at 300K and 340K iteration. The input image is resized to 736x736 for both
training and inference. We can achieve about 3x-higher FPS compared to FCPose
(ResNet-50), still showing higher accuracy of 64.2 APkp.

Fig. 5 illustrates the inference speed by the number of instances in an image.
Our MDPose shows the robust inference speed, regardless of the number of
people, even faster than FCPose. Furthermore, our MDPose with a heavier
backbone ResNet-101 surpasses FCPose with ResNet-50 regarding the inference
speed. It shows a strong potential of MDPose for the practical application
enabling real-time multi-person pose estimation.

Table 4: Comparisons with SOTA methods on OCHuman val/test set. The evaluation
metric is APkp.