Ed Greening111The workshop was supported by the University of Manchester,
IPPP, STFC, and IOP
on behalf of the LHCb collaboration

The University of Oxford

> Studies of rare decays are an indirect probe of New Physics (NP). This
> document presents recent measurements of rare decays in the charm sector by
> the LHCb experiment. The analyses are performed with proton-proton collision
> data at $\sqrt{s}$ = 7 ${\mathrm{\,Te\kern-1.00006ptV\\!}}$ recorded in
> 2011.

> PRESENTED AT
>
>
>
>
> The 6th International Workshop on Charm Physics
> (CHARM 2013)
> Manchester, UK, 31 August – 4 September, 2013

##  1 Introduction

Flavour-changing neutral current (FCNC) processes are rare within the Standard
Model (SM) as they cannot occur at tree level. At loop level, they are
suppressed by the both the Glashow-Iliopoulos-Maiani (GIM) [1] and the
Cabibbo-Kobayashi-Maskawa (CKM) [2, 3] mechanisms but are nevertheless well
established in processes that involve $K$ and $B$ mesons. In contrast to the
$B$ meson system, where the near-unity value of $|V_{ub}|$ and very high mass
of the top quark in the loop weaken the suppression, the cancellation is
almost exact in $D$ meson decays leading to lower SM branching fractions
($\cal B$). This suppression provides a unique opportunity to probe the
effects of NP on the coupling of up-type quarks in electroweak processes. NP
models may introduce additional diagrams that a priori need not be suppressed
in the same manner as the SM contributions. Enhancement in the $\cal B$ of
such decays would therefore be a sign of NP. The large number of $D$ mesons
created at the LHC and LHCb’s excellent ability to discriminate between pions
and muons [4, 5] mean that the detector is in a outstanding position to
investigate rare charm decays.

##  2 $D^{0}\rightarrow\mu^{+}\mu^{-}$

The decay $D^{0}\rightarrow\mu^{+}\mu^{-}$ is very rare in the SM because of
additional helicity suppression. The short distance perturbative contribution
to the $\cal B$ is of the order of $10^{-18}$ while the long distance non-
perturbative contribution, dominated by the two-photon intermediate state, is
estimated to be of the order $10^{5}$ higher [6]. A search for the decay is
performed with 0.9 $\mathrm{\,fb^{-1}}$ of data [7]. By taking the $D^{0}$
from $D^{+*}\rightarrow D^{0}\pi^{+}$ decays, a two-dimensional fit is
performed in m($\mu^{+}\mu^{-}$) and $\Delta m$ ($\equiv
m(\pi^{+}\mu^{+}\mu^{-})-m(\mu^{+}\mu^{-})$). The measured $\cal B$ is
normalised with the decay $D^{0}\rightarrow\pi^{+}\pi^{-}$. Peaking
backgrounds from the misidentified hadronic decays
$D^{0}\rightarrow\pi^{+}\pi^{-}$ and $D^{0}\rightarrow K^{-}\pi^{+}$ and the
misidentified and partially reconstructed semileptonic decay
$D^{0}\rightarrow\pi^{-}\mu^{+}\nu_{\mu}$ are also taken into account. The
observed number of events is consistent with the background expectations and
corresponds to an upper limit of $\cal B$($D^{0}\rightarrow\mu^{+}\mu^{-}$)
$<6.2\,(7.6)\times 10^{-9}$ at 90% (95%) CL. This represents an improvement of
more than a factor twenty with respect to previous measurements [8] but
remains several orders of magnitude larger than the SM prediction.

##  3 $D_{(s)}^{+}\rightarrow\pi^{+}\mu^{+}\mu^{-}$

The decay $D_{(s)}^{+}\rightarrow\pi^{+}\mu^{+}\mu^{-}$ proceeds via short and
long distance contributions. The long-distance contributions are mediated by
intermediate resonances,
$D_{(s)}^{+}\rightarrow\pi^{+}(V\rightarrow\mu^{+}\mu^{-})$, where $V$ $=$
$\phi$, $\eta$, $\rho^{0}$ or $\omega$, whose large $\cal B$ mask any
deviation from the much smaller non-resonant SM prediction, caused by NP. A
search for the decay is performed with 1.0 $\mathrm{\,fb^{-1}}$ of data [9].
The data is binned in m($\mu^{+}\mu^{-}$) allowing the long and short distance
contributions to be separated. The binning definitions are shown in Table 1.
The contribution from the intermediate $\rho^{0}$ and $\omega$ resonances are
grouped together as it is non-trivial to separate them. The signal yields in
each bin are determined with a simultaneous fit to the
m($\pi^{+}$$\mu^{+}\mu^{-}$) distribution of the m($\mu^{+}\mu^{-}$) bins and
shown in Table 1. The parameters of the shapes defining the $D_{(s)}^{+}$
signals are determined simultaneously across all bins. Candidates from the
kinematically similar $D_{(s)}^{+}\rightarrow\pi^{+}\pi^{+}\pi^{-}$ decay form
an important peaking background. Data-driven methods are used to parameterise
their contributions. The observed data, away from resonant structures, is
compatible with the background-only hypothesis, and no enhancement is
observed. Upper limits in the low and high m($\mu^{+}\mu^{-}$) bins are
calculated by normalising with the $\phi$ resonances. The upper limits in the
low and high m($\mu^{+}\mu^{-}$) bins, assuming a phase space $\mu^{+}\mu^{-}$
distribution, are extrapolated across the entirety of m($\mu^{+}\mu^{-}$) by
taking into account the relative efficiencies in each bin. Upper limits on the
non-resonant signal component of $\cal
B$($D^{+}\rightarrow\pi^{+}\mu^{+}\mu^{-}$) $<7.3\,(8.3)\times 10^{-8}$ and
$\cal B$($D_{s}^{+}\rightarrow\pi^{+}\mu^{+}\mu^{-}$) $<4.1\,(4.8)\times
10^{-9}$ at 90% (95%) CL are set. These represent an improvement of a factor
50 with respect to the previous limits [10, 11], but $\cal
B$($D^{+}\rightarrow\pi^{+}\mu^{+}\mu^{-}$) is still an order of magnitude
larger than the SM prediction.

Table 1: Signal yields for the $D_{(s)}^{+}\rightarrow\pi^{+}\mu^{+}\mu^{-}$ fits. Bin description |  m($\mu^{+}\mu^{-}$) range [${\mathrm{\,Me\kern-0.80005ptV\\!/}c^{2}}$] |  $D$ yield |  $D_{s}^{+}$ yield
---|---|---|---
low-m($\mu^{+}\mu^{-}$) | $\phantom{1}250-\phantom{1}525$ | $\phantom{00}-3\pm 11$ | $\phantom{-000}1\pm\phantom{0}6$
$\eta$ | $\phantom{1}525-\phantom{1}565$ | $\phantom{-00}29\pm\phantom{0}7$ | $\phantom{-00}22\pm\phantom{0}5$
$\rho^{0}/\omega$ | $\phantom{1}565-\phantom{1}850$ | $\phantom{-00}96\pm 15$ | $\phantom{-00}87\pm 12$
$\phi$ | $\phantom{1}850-1250$ | $\phantom{-}2745\pm 67$ | $\phantom{-}3855\pm 86$
high-m($\mu^{+}\mu^{-}$) | $1250-2000$ | $\phantom{-00}16\pm 16$ | $\phantom{0}-17\pm 16$

##  4 $D^{0}\rightarrow\pi^{+}\pi^{-}\mu^{+}\mu^{-}$

The non-resonant component of the decay
$D^{0}\rightarrow\pi^{+}\pi^{-}\mu^{+}\mu^{-}$ has an expected SM $\cal B$ of
the order $10^{-9}$ [12]. The branching fraction for these decays is expected
to be dominated by long-distance contributions involving resonances, such as
$D^{0}\rightarrow\pi^{+}\pi^{-}(V\rightarrow\mu^{+}\mu^{-})$, where $V$ can be
any of the light mesons $\eta$, $\rho^{0}$, $\omega$ or $\phi$. The
corresponding branching fractions can reach O($10^{-6}$) [12]. A search for
the decay is performed with 1.0 $\mathrm{\,fb^{-1}}$ of data [13]. The
$D^{0}\rightarrow\pi^{+}\pi^{-}\mu^{+}\mu^{-}$ data are split into four
m($\mu^{+}\mu^{-}$) bins: two bins containing the $\rho/\omega$ and $\phi$
resonances and two signal bins. No $\eta$ bin is defined due to a lack of
events after the analysis’s offline selection. The bin definitions are shown
in Table 2. By taking the $D^{0}$ from $D^{+*}\rightarrow D^{0}\pi^{+}$
decays, a two-dimensional fit is performed in m($\mu^{+}\mu^{-}$) and $\Delta
m$ ($\equiv
m(\pi^{+}_{s}\pi^{+}\pi^{-}\mu^{+}\mu^{-})-m(\pi^{+}\pi^{-}\mu^{+}\mu^{-})$)
in each bin. $D^{0}\rightarrow\pi^{+}\pi^{-}\pi^{+}\pi^{-}$ forms an important
peaking background and data-driven methods are used to parameterise the
contribution of this misidentified decay in each bin. The $\Delta m$ and
m($\pi^{+}\pi^{-}\mu^{+}\mu^{-}$) fits can be seen in Fig. 1 and Fig. 2,
respectively. The fitted yields are provided in Table 2. The observed data,
away from resonant structures, in both the low and high m($\mu^{+}\mu^{-}$)
bins, is compatible with the background-only hypothesis, and no enhancement is
observed. Upper limits in the low and high m($\mu^{+}\mu^{-}$) bins are
calculated by normalising with $\cal
B$($D^{0}\rightarrow\pi^{+}\pi^{-}(\phi\rightarrow\mu^{+}\mu^{-})$). The
normalisation $\cal B$ is estimated with the results of the amplitude analysis
of the $D^{0}\rightarrow K^{+}K^{-}\pi^{+}\pi^{-}$ decay performed at CLEO
[14] and the known value of $\cal B$($\phi$ $\rightarrow$
$\mu^{+}\mu^{-}$)$/$$\cal B$($\phi$ $\rightarrow$ $K^{+}K^{-}$) [15]. The
upper limits in the low and high m($\mu^{+}\mu^{-}$) bins, assuming a phase
space $\mu^{+}\mu^{-}$ distribution, are extrapolated across the entirety of
m($\mu^{+}\mu^{-}$) by taking into account the relative efficiencies in each
bin. An upper limit on the non-resonant signal component of $\cal
B$($D^{0}\rightarrow\pi^{+}\pi^{-}\mu^{+}\mu^{-}$) $<5.5\,(6.7)\times 10^{-7}$
at 90% (95%) CL is set.

Table 2: Signal yields for the $D^{0}\rightarrow\pi^{+}\pi^{-}\mu^{+}\mu^{-}$ fits. Bin description |  m($\mu^{+}\mu^{-}$) range [${\mathrm{\,Me\kern-0.80005ptV\\!/}c^{2}}$] |  $D^{0}\rightarrow\pi^{+}\pi^{-}\mu^{+}\mu^{-}$ yield
---|---|---
low-m($\mu^{+}\mu^{-}$) | $\phantom{1}250-\phantom{1}525$ | $\phantom{0}2\pm\phantom{0}2$
$\rho/\omega$ | $\phantom{1}565-\phantom{1}950$ | $23\pm\phantom{0}6$
$\phi$ | $\phantom{1}950-1100$ | $63\pm 10$
high-m($\mu^{+}\mu^{-}$) | $>1100$ | $\phantom{0}3\pm\phantom{0}2$

Figure 1: Distributions of $\Delta m$ for
$D^{0}\rightarrow\pi^{+}\pi^{-}\mu^{+}\mu^{-}$ decays in the (a)
low-m($\mu^{+}\mu^{-}$), (b) $\rho^{0}/\omega$, (c) $\phi$, and (d)
high-m($\mu^{+}\mu^{-}$) bins, with the $D^{0}$ invariant mass in the range
$1840-1888$ ${\mathrm{\,Me\kern-1.00006ptV\\!/}c^{2}}$. The data are shown as
points (black) and the fit result (dark blue line) is overlaid. The components
of the fit are also shown: the signal (black double-dashed double-dotted
line), the peaking background (green dashed line) and the non-peaking
background (red dashed-dotted line).

Figure 2: Distributions of m($\pi^{+}\pi^{-}\mu^{+}\mu^{-}$) for
$D^{0}\rightarrow\pi^{+}\pi^{-}\mu^{+}\mu^{-}$ decays in the (a)
low-m($\mu^{+}\mu^{-}$), (b) $\rho^{0}/\omega$, (c) $\phi$, and (d)
high-m($\mu^{+}\mu^{-}$) bins, with $\Delta m$ in the range $144.4-146.6$
${\mathrm{\,Me\kern-1.00006ptV\\!/}c^{2}}$. The data are shown as points
(black) and the fit result (dark blue line) is overlaid. The components of the
fit are also shown: the signal (black double-dashed double-dotted line), the
peaking background (green dashed line) and the non-peaking background (red
dashed-dotted line).

##  5 Conclusion

Before the second long shutdown of the LHC in 2017, LHCb expects to record an
additional 5 $\mathrm{\,fb^{-1}}$ at $\sqrt{s}$ = 13
${\mathrm{\,Te\kern-1.00006ptV\\!}}$. This is in addition to the 1 and 2
$\mathrm{\,fb^{-1}}$ of data at $\sqrt{s}$ = 7 and 8
${\mathrm{\,Te\kern-1.00006ptV\\!}}$, respectively, that LHCb already has on
tape. Together with anticipated improvements in LHCb’s trigger system and
analysis strategies, the higher centre-of-mass energy increases heavy flavour
production cross-sections. In comparison to the analyses detailed in this
report, a factor of twenty increase in the number of observed decays can
optimistically be hoped for. A naive $\sqrt{20}$ scaling, would then give the
following limits: $\cal B$($D^{0}\rightarrow\mu^{+}\mu^{-}$) $=1\times
10^{-8}$, an order of magnitude above the indirect bound; $\cal
B$($D_{(s)}^{+}\rightarrow\pi^{+}\mu^{+}\mu^{-}$) $=2\times 10^{-8}$, an order
of magnitude above the SM expectation; and $\cal
B$($D^{0}\rightarrow\pi^{+}\pi^{-}\mu^{+}\mu^{-}$) $=2\times 10^{-7}$, two
orders of magnitude above the SM expectation. So although one would not expect
to observe a SM signal before the LHCb upgrade, the phase space available to
NP is set to be further probed.

## References

  * [1] S. L. Glashow, J. Iliopoulos and L. Maiani, Phys. Rev. D 2 (1970) 1285
  * [2] N. Cabibbo, Phys. Rev. Lett. 10, 531-533 (1963)
  * [3] M. Kobayashi, T. Maskawa, Progress of Theoretical Physics 49 (2): 652-657 (1973)
  * [4] A. A. Alves Jr. et al. [LHCb collaboration], JINST 3 (2008) S08005.
  * [5] M. Adinolfi et al., Eur. Phys. J. C73 (2013) 2431
  * [6] G. Burdman, E. Golowich, J. L. Hewett and S. Pakvasa, Phys. Rev. D 66 (2002) 014009
  * [7] R. Aaij et al. [LHCb Collaboration], Phys. Lett. B 725 (2013) 15
  * [8] M. Petric et al. [Belle collaboration], Phys. Rev. D81 (2010) 091102
  * [9] R. Aaij et al. [LHCb Collaboration], Phys. Lett. B 724 (2013) 203
  * [10] V. M. Abazov et al. [D0 Collaboration], Phys. Rev. Lett. 100 (2008) 101801
  * [11] J. M. Link et al. [FOCUS Collaboration], Phys. Lett. B 572 (2003) 21
  * [12] L. Cappiello, O. Cata and G. D’Ambrosio, JHEP 1304 (2013) 135
  * [13] R. Aaij et al. [LHCb Collaboration], LHCb-PAPER-2013-050 submitted to PLB
  * [14] M. Artuso et al. [CLEO Collaboration], Phys. Rev. D 85 (2012) 122002
  * [15] J. Beringer et al. [Particle Data Group], Phys. Rev. D86 (2012) 010001


# KITRO: Refining Human Mesh by 2D Clues and Kinematic-tree Rotation

Fengyuan Yang Kerui Gu Angela Yao
National University of Singapore
{fyang, keruigu<EMAIL_ADDRESS>
###### Abstract

2D keypoints are commonly used as an additional cue to refine estimated 3D
human meshes. Current methods optimize the pose and shape parameters with a
reprojection loss on the provided 2D keypoints. Such an approach, while simple
and intuitive, has limited effectiveness because the optimal solution is hard
to find in ambiguous parameter space and may sacrifice depth. Additionally,
divergent gradients from distal joints complicate and deviate the refinement
of proximal joints in the kinematic chain. To address these, we introduce
Kinematic-Tree Rotation (KITRO), a novel mesh refinement strategy that
explicitly models depth and human kinematic-tree structure. KITRO treats
refinement from a bone-wise perspective. Unlike previous methods which perform
gradient-based optimizations, our method calculates bone directions in closed
form. By accounting for the 2D pose, bone length, and parent joint’s depth,
the calculation results in two possible directions for each child joint. We
then use a decision tree to trace binary choices for all bones along the human
skeleton’s kinematic-tree to select the most probable hypothesis. Our
experiments across various datasets and baseline models demonstrate that KITRO
significantly improves 3D joint estimation accuracy and achieves an ideal 2D
fit simultaneously. Our code available at: https://github.com/MartaYang/KITRO.

##  1 Introduction

(a) Reprojection and depth error after 2D keypoint refinement 222CLIFF [21]
proposes both a base model and a 2D keypoint refinement; we distinguish the
two as ‘CLIFFb’ and ‘CLIFFr’..

(b) Improvement of joints along the kinematic chain of the arm.

Figure 1:  (a) Our method has the lowest reprojection and depth errors. (b)
Competing methods exhibit a decrease in improvement and even deterioration
progressing along the kinematic chain from the hand to the spine; our approach
exhibits the opposite but maintains a significant and competitive improvement
for all joints.

3D human pose and shape estimation, also called Human Mesh Recovery (HMR), is
relevant for augmented and virtual reality applications. Statistical human
shape models like SMPL [24] have greatly simplified the HMR task. However,
state-of-the-art methods [12, 16, 21]

still suffer from the “misalignment problem” [18] where the predicted 3D mesh
does not align well with the 2D image evidence. The most advanced HMR model
[21] still has a 24 pixels reprojection error on the 3DPW dataset [35], as
shown in Fig. 2. On the other hand, estimating 2D human pose is more advanced
[3, 36, 39] and yields robust 2D keypoints, even under challenging scenarios
such as occlusion, varying lighting, and extreme poses. As such, a standard
strategy for improving HMR is to leverage 2D keypoints as a cue to refine the
estimated 3D meshes. Prior methods [2, 12, 21] optimize the pose and shape
parameters with a reprojection loss on the 2D keypoints with a standard
gradient descent. Such an approach, while intuitive and straightforward, is
not always effective.

A primary reason is the inherent depth ambiguity in 2D projection, as multiple
poses and shapes can fit the same 2D evidence. As such, optimizing solely on
2D reprojection does not account for depth ambiguity so is unlikely to find
the optimal solution and may even increase the depth error (see HMR+SMPLify
[2] in Fig. 2). Secondly, existing methods optimize all the body joints
collectively through gradient descent. Yet the gradient updates at different
joints may be incongruent. Updates at the distal joints far down the kinematic
chain, such as the wrist or hands, are backpropagated to the proximal joints
closer to the root, like the shoulder or collar. Divergent gradients can
complicate the update of proximal joints, limiting the refinement
improvements, to the extent that it may even harm the overall accuracy (see
Fig. 1(b)).

In this work, we offer the key insight that joint depth can be solved
explicitly in closed-form. Provided with 2D keypoints, the depth of the parent
joint, and the 3D length of the bone connecting the two joints, the problem
can be narrowed down into two solutions. Of these two solutions, one
corresponds to the bone pointing toward the camera, and the other corresponds
to the bone pointing away from the camera as shown in Fig. 3. In this way, the
depth ambiguity in the solution space can be largely reduced.

Having two possible depths for every joint naturally forms a binary tree
progressing along kinematic chains in the human body (see Fig. 4). Any path
traversing the tree is a hypothesis and we can formulate overall pose
refinement as a selection problem of finding an optimal path. Such a strategy
has a distinct advantage in that it can equally improve proximal and distal
joints.

Based on these insights, we propose a novel plug-and-play human mesh
refinement strategy which we call Kinematic-tree Rotation (KITRO). KITRO
explicitly models joint depth and solves for bone direction as a swing
rotation in closed form, ensuring an excellent fit to 2D keypoints with lower
depth errors. KITRO employs a decision tree to handle divergences among joints
in the kinematic-tree, effectively tracing and selecting the most probable
hypotheses for stable and consistent improvements across the kinematic chain.
Our experimental results, on various datasets and base models, demonstrate
significantly higher accuracy compared to existing methods.

We highlight our key contributions as follows:

  * •

KITRO’s explicit depth modeling and closed-form calculation diminish
ambiguities in solution space, enhancing depth accuracy and obtaining ideal 2D
fit simultaneously.

  * •

KITRO’s decision-tree-based hypotheses tracing and selection for joint
rotations encompasses the entire kinematic-tree, benefiting both proximal and
distal joints.

  * •

Our method’s effectiveness, especially large gains in pose accuracy, is
validated across datasets and base models.

##  2 Related Works

Human Mesh Recovery (HMR). HMR methods are either non-parametric, in that they
directly estimate the 3D mesh vertices [33, 40, 25, 30, 23, 22] or parametric,
using statistical models [13, 18, 29, 31, 21, 4]. In using a statistical model
like SMPL [24], the HMR task simplifies to an estimation on the model
parameters. However, estimating the parameters can be highly challenging under
monocular settings despite the advancements in network architecture [38, 32,
34] and learning paradigms [20, 26, 27]. More recently, there are also hybrid
approaches combining parametric and non-parametric [19].

Human Mesh Refinement with 2D Keypoints. One curious observation is that
estimated meshes are often poorly aligned to the 2D image evidence. As such,
optimization methods have been proposed to leverage 2D keypoints as an
additional cue to refine estimated 3D human meshes. A simple and intuitive way
for refinement is simply to update the mesh model parameters, _i.e_. the SMPL
parameters with an additional 2D reprojection loss [2, 18, 9]. However, as 2D
image evidence is an ambiguous cue for 3D estimates, directly update the SMPL
parameters tends to result in unnatural poses, even if they are better aligned
to the provided 2D keypoints. As such, methods like EFT [12] and CLIFF [21]
use the 2D reprojection loss to update the HMR estimation network weights
instead. Such a refinement, however, is dataset-specific and thus loses
generalization. Additionally, finding ways to avoid overfitting or
underfitting remains a challenging limitation in these fine-tuning methods.
Our work also focuses on mesh refinement; however, instead of optimization
with a 2D reprojection loss, we opt to solve for the refined solution in
closed form.

##  3 Preliminaries

SMPL. The SMPL model [24] is a commonly used 3D statistical shape model of the
human body. It maps pose parameters $\boldsymbol{\theta}\in\mathbb{R}^{P\times
3}$ and shape parameters $\boldsymbol{\beta}\in\mathbb{R}^{10}$ to a 3D mesh
of the body $\mathbf{V}\\!\in\\!\mathbb{R}^{N\times 3}$, where $N\\!=\\!6890$
and $P\\!=\\!24$ are the number of vertices and body joints respectively. We
denote the
$\mathbf{J}(\boldsymbol{\beta},\boldsymbol{\theta})\in\mathbb{R}^{P\times 3}$
as the function that maps pose parameters $\boldsymbol{\theta}$ and shape
parameters $\boldsymbol{\beta}$ to joint locations. According to human
kinematic-tree shown in Fig. 4, we define the Euclidean distance between child
joint $\mathbf{J}^{c}(\boldsymbol{\beta},\boldsymbol{\theta})$ and its parent
joint $\mathbf{J}^{p}(\boldsymbol{\beta},\boldsymbol{\theta})$ as the bone
length:

$bl_{3D}^{(p,c)}=\left\|\mathbf{J}^{p}(\boldsymbol{\beta},\boldsymbol{\theta})-\mathbf{J}^{c}(\boldsymbol{\beta},\boldsymbol{\theta})\right\|_{2}.$
(1)

Swing-Twist Decomposition. The pose parameters $\boldsymbol{\theta}$ are the
relative rotation of each joint, where the rotation of the pelvis 333Also
known as root joint, we use them interchangeably in this paper. joint
$\boldsymbol{\theta}^{0}$ serves as the global rotation of the human body. The
standard SMPL model uses the axis-angle
$\\{\boldsymbol{\theta}^{0},\cdots,\boldsymbol{\theta}^{23}\\}$ to represent
the rotation of each joint and rotates the joints along the kinematic chain
from the pelvis to the end joints. An alternative [19] proposes to represent
each joint rotation in rotation matrix form
$\boldsymbol{\theta}_{R}^{i}\in\mathbb{SO}(3)$ with the swing-twist
decomposition: $\boldsymbol{\theta}_{R}^{i}=R_{sw}R_{tw}$, where
$\boldsymbol{\theta}_{R}^{i}$ is equivalent rotation matrix form of
$\boldsymbol{\theta}^{i}$. Given the bone direction, the swing rotation which
is 2 degrees of freedom can be derived in closed form from Rodrigues’ formula
(full details in Sec. A of the Supplementary).

Human Mesh Recovery and Refinement. Monocular HMR models take an image
$\mathbf{X}\in\mathbb{R}^{H\times W}$ as input and predicts SMPL parameters
$({\boldsymbol{\theta}},{\boldsymbol{\beta}})$ and the camera translation
${\mathbf{t}}\in\mathbb{R}^{3}$ as part of the camera extrinsics to project
the 3D mesh onto the image plane. Mesh refinement methods [2, 12, 21] uses
additional 2D keypoints $\mathbf{j}\in\mathbb{R}^{P\times 2}$ to refine some
estimated SMPL and camera translation parameters
$(\hat{\boldsymbol{\theta}},\hat{\boldsymbol{\beta}},\hat{\mathbf{t}})$. The
refinement process can be formalized as:

$\left(\hat{\boldsymbol{\theta}}^{\prime},\hat{\boldsymbol{\beta}}^{\prime},\hat{\mathbf{t}}^{\prime}\right)=\text{Refine}(\hat{\boldsymbol{\theta}},\hat{\boldsymbol{\beta}},\hat{\mathbf{t}},\mathbf{j}).$
(2)

The refinement as given in Eq. 2 can also be applied iteratively such as
SMPLify [2], by using the refined outputs
$(\hat{\boldsymbol{\theta}}^{\prime},\hat{\boldsymbol{\beta}}^{\prime},\hat{\mathbf{t}}^{\prime})$
as the input estimates to continue the refinement process. A commonly used
approach for refinement is to minimize the 2D reprojection loss of the
estimated 3D joints ${\mathbf{J}(\boldsymbol{\beta},\boldsymbol{\theta})}$
with respect the provided 2D joints $\mathbf{j}$:

$\mathcal{L}_{j2D}=\min_{\boldsymbol{\theta},\boldsymbol{\beta},\mathbf{t}}\left\|\pi\left(\mathbf{J}(\boldsymbol{\beta},\boldsymbol{\theta})+\mathbf{t}\right)-\mathbf{j}\right\|_{2},$
(3)

where $\pi$ indicates the camera projection function. These parameters can be
optimized jointly [12, 21] or fix some parameters and optimize others [2]. It
is worth mentioning that when fixing $\boldsymbol{\theta}$ and
$\boldsymbol{\beta}$, the optimal $\mathbf{t}$ can be solved in closed-form
because it can be formulated as a least squares problem.

Camera Intrinsics and Extrinsics. In our work, we relax the weakly-perspective
assumption and adopt a full-perspective camera model like prior studies [14,
17, 37]. Consistent with these works [17, 21], we estimate the camera focal
length as $f=\sqrt{H^{2}+W^{2}}$. Therefore, the camera intrinsic can be
formulated as:

$K=\begin{bmatrix}f&0&W/2\\\ 0&f&H/2\\\ 0&0&1\end{bmatrix}.$ (4)

Following the convention in HMR [13, 18, 15, 6, 16], the camera rotation is
set as $I$ and absorbed into the global orientation of the human (_i.e_.,
$\boldsymbol{\theta}^{0}$) predicted by the HMR model. Thus, the translation
$\mathbf{t}$ is the only camera extrinsic that needs to be estimated.

##  4 Method

As illustrated in Fig. 2, our refinement framework iteratively updates the
estimates for the camera and shape (Sec. 4.1) as well as the pose (Sec. 4.2,
Sec. 4.3). Unlike previous works [2, 12, 21], which update pose and shape
parameters jointly, we perform an individual update while keeping the others
fixed based on the previously refined value. This allows us to focus on the
pose refinement where we explicitly model the depth in a closed-form manner
and choose the most likely hypothesis by using a decision tree along the
kinematic-tree.

Figure 2: Our framework overview. Starting with an initial human mesh and 2D
keypoints, our iterative refinement operates over camera, shape, and pose. The
process involves localizing camera translation for 2D-3D joint alignment,
optimizing shape for more aligned bone length, and generating and selecting
bone direction hypotheses for rotation refinement along the kinematic-tree.

###  4.1 Camera and Shape Refinement

As can be seen in Fig. 2, the given 2D keypoints together with the current
human mesh can definitely provide some information and constraints for the
camera translation and human shape. These constraints can help us to refine
these two factors better.

Camera Translation Adjustment. We estimate the camera translation based on the
2D reprojection loss given by Eq. 3. Minimizing the loss is equivalent to
solving a least-squares optimization for the camera translation [18]:

$\mathbf{t}^{*}=\operatorname*{arg\,min}_{\mathbf{t}}\left\|\pi(\mathbf{J}(\boldsymbol{\beta},\boldsymbol{\theta})+\mathbf{t})-\mathbf{j}\right\|_{2},$
(5)

where $\mathbf{t}^{*}$ can be found with SVD. Note that $\mathbf{t}^{*}$ is
the optimal solution for a given $\boldsymbol{\theta}$ and
$\boldsymbol{\beta}$.

From an iterative update perspective, we find it more effective to update
$\mathbf{t}$ with a moving average. For current camera translation
$\mathbf{t}$, the updated $\mathbf{t}^{\prime}$ is given as:

$\mathbf{t}^{\prime}=(\mathbf{t}^{*}+\mathbf{t})/2.$ (6)

The reason for Eq. 6 is that $\mathbf{t}^{*}$ can be affected by the noise of
$\boldsymbol{\theta}$ and $\boldsymbol{\beta}$. In that sense, the moving
average keeps the historical information from the original HMR prediction,
acting like a good regularizer.

Shape Optimization. Previous works update $\boldsymbol{\beta}$ based on the 2D
reprojection of the individual joints
$\mathbf{J}(\boldsymbol{\beta},\boldsymbol{\theta})$. We consider the
refinement of $\boldsymbol{\beta}$ from a _bone length_ perspective. Consider
a parent joint indexed by $p$, with estimated 3D joint
$\mathbf{J}^{p}(\boldsymbol{\beta},\boldsymbol{\theta})$ and provided 2D joint
$\mathbf{j}^{p}$; similarly, for a child joint indexed by $c$, consider
$\mathbf{J}^{c}(\boldsymbol{\beta},\boldsymbol{\theta})$. The projected 2D
bone length between the $(p,c)$ joint pair is defined as the Euclidean
distance between the two joints:

$bl^{(p,c)}_{proj}(\boldsymbol{\beta})=\left\|\pi(\mathbf{J}^{p}(\boldsymbol{\beta},\boldsymbol{\theta})+\mathbf{t}^{\prime})-\pi(\mathbf{J}^{c}(\boldsymbol{\beta},\boldsymbol{\theta})+\mathbf{t}^{\prime})\right\|_{2}.$
(7)

For clarity, we specify that the projected bone length
$bl^{(p,c)}_{proj}(\boldsymbol{\beta})$ depends only on $\boldsymbol{\beta}$,
as we treat $\boldsymbol{\theta}$ and $\mathbf{t}^{\prime}$ as fixed constants
while we update the shape parameter $\boldsymbol{\beta}$. The bone length of
the provided 2D joints can be defined similarly as

$bl_{2D}^{(p,c)}=\left\|\mathbf{j}^{p}-\mathbf{j}^{c}\right\|_{2}.$ (8)

To estimate the shape loss, we consider an L1-norm between the projected bone
length in Eq. 7 and the given bone lenght of Eq. 8 over all the bones or
${p,c}$ combinations in the human body:

$L(\boldsymbol{\beta})=\sum_{p,c}\left|bl^{(p,c)}_{proj}(\boldsymbol{\beta})-bl_{2D}^{(p,c)}\right|.$
(9)

Unlike the camera parameter $\mathbf{t}$, the optimal shape parameter
$\boldsymbol{\beta}$ cannot be solved for in closed form. As such, we estimate
the updated shape parameter $\boldsymbol{\beta}^{\prime}$ with gradient-
descent based optimization of the loss in Eq. 9:

$\boldsymbol{\beta}^{\prime}=\boldsymbol{\beta}-\eta\nabla_{\boldsymbol{\beta}}L(\boldsymbol{\beta})\quad\text{
for }T\text{ steps},$ (10)

where $\eta$ is the learning rate and optimization takes T steps.

###  4.2 Pose Hypothesis Generation

3D Bone Direction Calculation.

Figure 3: Calculation of two bone direction solutions of the (p, c) joint
pair, based on 2D keypoints, bone length, and the depth of the parent joint.
One points towards the camera, and the other away.

To refine the pose parameters $\boldsymbol{\theta}$, we opt to work with the
swing-twist decomposed form of the joint rotation, rather than in the standard
axis-angle representations of SMPL. This is key in our refinement approach
because it allows us to estimate the swing angle in closed form directly from
the provided 2D keypoints. Consider the 2D keypoints $\mathbf{j}^{p}$ and
$\mathbf{j}^{c}$ of a bone $(p,c)$ as defined in the SMPL kinematic tree.
Their corresponding 3D joints
$\mathbf{J}^{p}(\boldsymbol{\beta}^{\prime},\boldsymbol{\theta})$ and
$\mathbf{J}^{c}(\boldsymbol{\beta}^{\prime},\boldsymbol{\theta})$ are located
along the rays $\vec{p}$ and $\vec{c}$ from the camera to the 2D keypoints on
the image plane. And $O$ denotes the camera location which is the negative of
camera translation $O=-\mathbf{t}^{\prime}$ since camera rotation is identity.
As shown in Fig. 3, directions of these two rays are:

$\vec{p}=K^{-1}\times\mathbf{j}^{p}_{h},\quad\vec{c}=K^{-1}\times\mathbf{j}^{c}_{h},$
(11)

where subscript $h$ denotes homogenized coordinates and $K$ is the camera
intrinsic matrix defined in Sec. 3. Based on the direction of the rays
$\vec{p}$ and $\vec{c}$, their intersection angle $\alpha$ satisfies the
following:

$\cos\alpha=\frac{\vec{p}\cdot\vec{c}}{\lVert\vec{p}\rVert\lVert\vec{c}\rVert},\quad\sin\alpha=\frac{\lVert\vec{p}\times\vec{c}\rVert}{\lVert\vec{p}\rVert\lVert\vec{c}\rVert}.$
(12)

Now, as we know the 3D bone length $bl_{3D}^{(p,c)}$ defined in Eq. 1 and the
depth of the parent joint from the camera
$|O{J}^{p}|=\left\|\mathbf{J}^{p}(\boldsymbol{\beta}^{\prime},\boldsymbol{\theta})+\mathbf{t}^{\prime}\right\|_{2}$,
we can directly solve for the child joint in closed form. Specifically,
$|FJ^{c}_{0}|=|FJ^{c}_{1}|=\sqrt{\left(bl_{3D}^{(p,c)}\right)^{2}-\left(\left|OJ^{p}\right|\sin\alpha\right)^{2}}$
based on Fig. 3. Therefore, we can calculate the child joint $J^{c}$ specified
by the vector $\overrightarrow{J^{p}J^{c}}$ from the parent joint $J^{p}$, and
the depth $|OJ^{p}|$ from the camera $O$:

$\displaystyle\overrightarrow{J^{p}J^{c}}=\underbrace{|OJ^{p}|\cdot(\cos\alpha\cdot\vec{c}-\vec{p})}_{\overrightarrow{J^{p}F}}\pm\underbrace{|FJ^{c}_{0}|\cdot\vec{c}}_{\overrightarrow{FJ^{c}_{0}}},$
(13) $\displaystyle|OJ^{c}|={|OJ^{p}|\cdot\cos\alpha}\pm{|FJ^{c}_{0}|}.$

There are two possibilities for the child joint, as indicated by the $\pm$
sign in the solutions in Eq. 13. One solution points towards the camera
($\overrightarrow{J^{p}J^{c}_{0}}$ in Fig. 3) while the other points away
($\overrightarrow{J^{p}J^{c}_{1}}$). These two solutions directly illustrate
the inherent depth ambiguity. Depending on the accuracy of the estimated
terms, the square root term may become negative (might arise during decision
tree calculations in the next step); for numerical stability, we rectify it to
0.

From Bone Direction to Full Body Pose Hypothesis. Note that the solution for
the bone direction in Eq. 13 depends on the depth of the parent joint
$|OJ^{p}|$. As the parent joint’s depth has two possible solutions as well,
and the depth dependency propagates through the kinematic tree, the hypotheses
for possible poses naturally form binary trees. Fig. 4 shows an example of the
left leg, beginning at the pelvis root node, with child nodes for the hip,
knee, ankle, and toe. For the entire body pose, there are 5 trees,
representing the arms, legs, and torso. A full-body pose hypothesis is then
represented by a path through each of the trees. It’s worth mentioning that
while we could select solutions for each joint greedily, our approach
leverages a decision tree naturally formed along the kinematic chain since it
enables more accurate selections by considering all potential outcomes.

###  4.3 Pose Hypothesis Selection

Figure 4: Human kinematic-tree in SMPL [24] and ours decision tree example
tracing all hypotheses for leg joints. In our method, local solution
certainties are used as edge weights, with the largest global path product
representing the most probable pose.

Decision Tree Formulation. From the binary trees, we wish to select an optimal
path. Yet without additional information or cues, it is challenging to know
how optimality should be measured. As such, we rely only on the initial HMR
estimate, and compute the cosine similarity between the relative bone
rotations for the two closed-form solutions
$R_{rel}(\overrightarrow{J^{p}J^{c}_{k}}|\phi(p))$ for $k=\\{0,1\\}$ and the
relative bone rotation predicted by the original HMR model
$\boldsymbol{\theta}_{\text{HMR}}^{p}$:

$\text{cos\\_sim}(\overrightarrow{J^{p}J^{c}_{k}}|\phi(p))=\frac{cos\left<R_{rel}(\overrightarrow{J^{p}J^{c}_{k}}|\phi(p)),\boldsymbol{\theta}_{\text{HMR}}^{p}\right>+1}{2}$
(14)

where $\phi(p)=(\phi_{0},\cdots,\phi_{i},\cdots,\phi_{p})$ represents the path
from the root to the current parent joint $p$. Each $\phi_{i}\in\\{0,1\\}$
denotes the two possible solutions for each bone with $\mathbf{J}_{i}$ as the
child joint along the chain, except $\phi_{0}$ since the root joint is fixed
after camera refinement for each iteration. We directly apply the Softmax of
these two cosine similarities as the weights for edges in the decision tree as
shown in Fig. 4:

$w(e^{(p,c)}_{k}|\phi(p))=\frac{\exp(\text{cos\\_sim}(\overrightarrow{J^{p}J^{c}_{k}}|\phi(p)))}{\sum_{k^{\prime}=0}^{1}\exp(\text{cos\\_sim}(\overrightarrow{J^{p}J^{c}_{k^{\prime}}}|\phi(p)))}.$
(15)

Intuitively, Eq. 15 measures the relative consistency between calculated and
original prediction. It relies on the assumption that the original HMR
prediction, although not precise at exact bone direction, is sufficiently
accurate at indicating whether the bone points towards or away from the
camera. We verify this assumption empirically in Sec. B of the Supplementary.

The computational complexity for calculating the solutions in the binary trees
depends only on the depth, as nodes within any given depth of the trees can be
computed in parallel. By constructing the binary trees and estimating the
weights, we obtain a global view on the pose feasibility. We choose as the
optimal path the one with the highest node products. The final pose is defined
by the optimal selection path:

$\phi^{*}=\operatorname*{arg\,max}_{\phi}{\prod_{(p,c)}w(e^{(p,c)}_{\phi_{c}}|\phi(p))},$
(16)

where $\phi=\\{\phi_{0},\cdots,\phi_{23}\\}$ is a pose hypothesis for all 23
bones of a human body.

Pose Parameter Update. We perform an update on current $\boldsymbol{\theta}$
according to the selection chain $\phi^{*}$ in Eq. 16. The update is soft,
weighted by the edge weight $w(e^{(p,c)}_{\phi^{*}_{c}}|\phi^{*}(p))$
calculated in Eq. 15 which acts like a certainty term for choosing
$\phi^{*}_{c}$ instead of the other (_i.e_., $1-\phi^{*}_{c}$):

$\lambda^{(p,c)}=w(e^{(p,c)}_{\phi^{*}_{c}}|\phi^{*}(p)).$ (17)

The final bone direction $\vec{\mathbf{b}}_{new}^{(p,c)}$ is updated as the
weighted sum of the selected solution
$\overrightarrow{{J^{p}J^{c}_{\phi_{c}^{*}}}}$ and current bone direction
$\vec{\mathbf{b}}^{(p,c)}=\mathbf{J}^{c}(\boldsymbol{\beta}^{\prime},\boldsymbol{\theta})-\mathbf{J}^{p}(\boldsymbol{\beta}^{\prime},\boldsymbol{\theta})$:

$\vec{\mathbf{b}}_{new}^{(p,c)}=\lambda^{(p,c)}\cdot\overrightarrow{{J^{p}J^{c}_{\phi_{c}^{*}}}}+(1-\lambda^{(p,c)})\cdot\vec{\mathbf{b}}^{(p,c)}.$
(18)

Finally, the pose parameter $\boldsymbol{\theta}^{p}$ of the parent joint $p$
can be updated based on the swing rotation $R_{sw}^{(p,c)}$ which rotates
$\vec{\mathbf{b}}^{(p,c)}$ to $\vec{\mathbf{b}}_{new}^{(p,c)}$ simply by the
Rodrigues’ rotation formula. In the special case of the root joint and the
third spine joint (_i.e_., ‘Spine3’), where there are three children denoted
as $c_{0},c_{1},c_{2}$, we compute a rotation matrix $R_{sw}^{(p,c)}$ that
optimally rotates the vectors
$\\{\vec{\mathbf{b}}^{(p,c_{0})},\vec{\mathbf{b}}^{(p,c_{1})},\vec{\mathbf{b}}^{(p,c_{2})}\\}$
to best align with
$\\{\vec{\mathbf{b}}_{new}^{(p,c_{0})},\vec{\mathbf{b}}_{new}^{(p,c_{1})},\vec{\mathbf{b}}_{new}^{(p,c_{2})}\\}$
by SVD [19]. Finally, by applying above rotation matrix $R_{sw}^{(p,c)}$ to
update parent joint $p$’s rotation $\boldsymbol{\theta}_{R}^{p}$, we obtain
the refined pose parameter:

${\boldsymbol{\theta}_{R}^{p}}^{\prime}=(\prod_{i\in
KC(\tilde{p})}\boldsymbol{\theta}_{R}^{i})^{T}\cdot
R_{sw}^{(p,c)}\cdot\prod_{i\in KC(p)}\boldsymbol{\theta}_{R}^{i},$ (19)

where $\tilde{p}$ is the parent joint of $p$, and $KC(p)$ is the kinematic
chain from root to joint $p$. Proof of correctness is given in Sec. C of the
Supplementary. Updating all joints by Eq. 19 along kinematic-tree gets the
refined $\boldsymbol{\theta}^{\prime}$.

The updates in Eq. 6, Eq. 10, and Eq. 19 specify one refinement iteration for
$\mathbf{t}$, $\boldsymbol{\beta}$ and $\boldsymbol{\theta}$. We can continue
to iterate by using the refined parameters as initial estimates for a total of
$M$ iterations.

###  4.4 Implementation Details

Here we elaborate on more implementation details. For the shape refinement, we
employ Adam optimizer to optimize Eq. 9 for $T=10$ steps with a learning rate
$\eta=0.1$ in each iteration. And the whole iteration number is $M=10$. Ours
overall pseudo-code is shown in Algorithm 1.

Algorithm 1 Ours Human Mesh Refinement

1:Initial pose $\boldsymbol{\theta}_{0}$, shape $\boldsymbol{\beta}_{0}$,
camera translation $\mathbf{t}_{0}$, 2D keypoints $\mathbf{j}$, iterations
$M$, kinematic-tree $KT$.

2:Refined $\boldsymbol{\theta}_{M}$, $\boldsymbol{\beta}_{M}$,
$\mathbf{t}_{M}$.

3:for $m=0\to M-1$ do

4: $\mathbf{t}^{*}\leftarrow$ Best $\mathbf{t}$ aligned $\mathbf{j}$ and
$\mathbf{J}(\boldsymbol{\beta}_{m},\boldsymbol{\theta}_{m})$ $\triangleright$
Eq. 5

5: $\mathbf{t}_{m+1}\leftarrow(\mathbf{t}^{*}+\mathbf{t}_{m})/2$
$\triangleright$ Eq. 6

6: $\boldsymbol{\beta}_{m+1}\leftarrow$ Adam optimize $\boldsymbol{\beta}_{m}$
by $L(\boldsymbol{\beta})$ $\triangleright$ Eq. 9

7: $DecisionTree\leftarrow$ Binary solutions along $KT$

8: $\phi^{*}\leftarrow$ Optimal path in $DecisionTree$ $\triangleright$ Eq. 16

9: $\boldsymbol{\theta}_{m+1}\leftarrow$ Update bone rotation base on
$\phi^{*}$ $\triangleright$ Eq. 19

10:end for

##  5 Experiments

###  5.1 Dataset and Metrics

3DPW dataset serves as a rigorous outdoor benchmark tailored for 3D pose and
shape estimation. We follow [21, 12, 11] and use the ground truth 2D keypoint
annotations from 3DPW as refinement inputs. Human3.6M is an indoor 3D human
mesh dataset. Consistent with preceding works [13, 18, 5], we use subjects S9
and S11 for testing. Similarly, we use the ground truth 2D keypoints for
refinement inputs.

We use three standard 3D pose and shape metrics: (1) MPJPE measures the
average Euclidean distance between the ground truth and the predicted joint
positions, only considering alignment at the pelvis, (2) PA-MPJPE which is the
MPJPE error after further aligning the predicted pose to the ground truth with
Procrustes aligned and (3) PVE measuring the average Euclidean distance
between the predicted and the ground truth mesh vertices after pelvis
alignment.

###  5.2 Ablation Study

We adopted CLIFFb [21] as the baseline HMR model. The ablation study upon our
full model for three refinement factors, camera, shape, and pose, are shown in
Tab. 1.

Table 1: Ablation study for camera, shape, and pose refinement on 3DPW across
three segments. The bottom segment presents our full model’s results. ‘DT’
stands for Decision Tree.

Method | PA-MPJPE $\downarrow$ | MPJPE $\downarrow$ | PVE $\downarrow$
---|---|---|---
fixed $\mathbf{t}$ as $\mathbf{t}_{0}$ | 28.53 | 46.68 | 57.76
hard-update $\mathbf{t}$ (Eq. 5) | 27.99 | 50.21 | 61.11
Large focal length | 32.72 | 52.46 | 63.61
fixed $\boldsymbol{\beta}$ as $\boldsymbol{\beta}_{0}$ | 35.00 | 69.02 | 84.25
$\boldsymbol{\beta}$ from $\mathcal{L}_{j2D}$ of Eq. 3 | 32.60 | 52.60 | 64.45
fixed $\boldsymbol{\theta}$ as $\boldsymbol{\theta}_{0}$ | 44.57 | 80.03 | 95.71
greedy + hard-update | 34.32 | 54.91 | 67.05
greedy + soft-update | 29.24 | 45.94 | 56.57
DT + hard-update | 32.18 | 52.15 | 63.82
KITRO (ours) | 27.67 | 43.53 | 53.44

Camera Refinement. The first segment in Tab. 1 shows the impact when we do not
refine the camera estimate and fix it to the HMR estimate (first row) and do
not use the moving average and directly replace the camera estimate according
to the optimal translation of Eq. 5 without Eq. 6 (second row). Performance
drop shows the effectiveness of the camera translation refinement and the
moving average as a good regularizer. The third row, using a fixed focal
length of 5000, shows an obvious performance drop, indicating the limitations
of a weak perspective camera model.

Shape Refinement. The second segment in Tab. 1 shows the impact when the shape
parameter $\boldsymbol{\beta}$ is not updated (first row in this segment) vs.
optimizing $\boldsymbol{\beta}$ according to the standard 2D projection loss
of the keypoints (second row) and our proposed loss on the 2D bone length
projection (as per Eq.9). This comparison reveals that our loss is more
effective than $\mathcal{L}_{j2D}$ in our method. The ablation for refinement
step $T$ and learning rate $\eta$ are given in Sec. D of the Supplementary,
they are all not highly sensitive factors.

Table 2: SOTA comparison on 3DPW and Human3.6M. Baseline HMR model for human
mesh initialization in the first segment, refinement methods in the second
segment including our results at the bottom. ∗ indicates our reproduced
results. † marks a different 3DPW evaluation protocol using extra gender
information in data preprocessing. All our reproductions ensure the same
protocol and fair comparison.

| 3DPW |  | Human3.6M
---|---|---|---
Method | PA-MPJPE $\downarrow$ | MPJPE $\downarrow$ | PVE $\downarrow$ |  | PA-MPJPE $\downarrow$ | MPJPE $\downarrow$
CLIFFb† [21] | 43.0 | 69.0 | 81.2 |  | - | -
CLIFFb∗ | 43.76 | 73.67 | 91.58 |  | 36.16 | 55.18
DynaBOA [11] | 40.4 | 65.5 | 82.0 |  | - | -
Pose2Mesh [5] | 34.6 | 65.1 | - |  | 35.3 | 51.1
CLIFFb + CLIFFr† [21] | 32.8 | 52.8 | 61.5 |  | - | -
CLIFFb + SMPLify∗ | 36.11 | 66.67 | 79.91 |  | 28.07 | 45.19
CLIFFb + CLIFFr∗ | 32.04 | 55.83 | 71.95 |  | 25.88 | 42.79
CLIFFb + KITRO (ours) | 27.67 (4.3$\downarrow$) | 43.53 (12.3$\downarrow$) | 53.44 (18.5$\downarrow$) |  | 21.04 (4.8$\downarrow$) | 34.50 (8.3$\downarrow$)

Pose Refinement. The third segment in Tab. 1 shows the impact when the pose
refinement is removed (first row in this segment) as well as alternative
designs on the decision tree (vs greedy selection, second and third row) and
the weighted update of Eq. 18 (vs hard-update, second and fourth row). The
decreased performance in these designs shows the global perspective offered by
our decision tree and the necessity of reweighting for a moderated update.

Parameter Impact. Across the three refined parameters, $\boldsymbol{\theta}$
has the biggest impact; removing it from the refinement leads to a
$\approx$60% increase in these three evaluation errors. In contrast,
$\boldsymbol{\beta}$ has a $\approx$30% increase, and the camera has the least
impact of $<1\%$. This demonstrates the crucial of pose and shape refinement
in our method, with the KITRO design for pose refinement notably enhancing
performance. Camera refinement, however, exhibits less sensitivity.

Figure 5: Impact of Refinement Iterations for MPJPE, PA-MPJPE, and MPVPE on
3DPW. Three line plots demonstrate a graceful decrease and quick convergence
of error metrics.

Iterative Updates. Fig. 5 plots the three evaluation metrics with respect to
the number of refinement iterations. The errors gracefully decrease with
increasing interactions, converging to the final result after 4-5 iterations.
Fig. 6 shows qualitative examples of body poses over the iterations. The
initial predictions are misaligned with the visual evidence from the image;
our method progressive refines the parameters to match the evidence, while
preserving natural poses.

Different HMR Models and Representation. Our method is plug-and-play on top of
any HMR method. We test our method on other commonly used base models such as
SPIN [18], EFT [12] and a different rotation representation PRoM [10]. As
shown in Tab. 3, our method can also make large improvements and outperform
previous refinement methods with large margins, especially in MPJPE and PVE
which are indicators for global orientation and shape.

Table 3: Results using alternative HMR base models and rotation
representations on 3DPW. ∗ indicates our reproduced results.

Method | PA-MPJPE $\downarrow$ | MPJPE $\downarrow$ | PVE $\downarrow$
---|---|---|---
SPIN [18] | 59.97 | 102.12 | 130.62
SPIN + SMPLify∗ | 47.99 | 87.06 | 102.28
SPIN \+ ours | 42.46 | 67.12 | 80.25
EFTb [12] | 54.71 | 94.02 | 116.23
EFTb + SMPLify∗ | 44.69 | 82.39 | 96.50
EFTb \+ ours | 32.34 | 49.14 | 59.28
PRoM [10] | 42.0 | 67.6 | 79.2
PRoM + SMPLify∗ | 35.67 | 65.21 | 78.02
PRoM \+ ours | 26.72 | 42.18 | 51.25

Figure 6: Refinement result over iterations (left part) and comparison with
other human mesh refinement methods (right part). Left: The blue circles
highlight the refinement progress where our method iteratively refines the
misaligned bone direction to fit with 2D. Right: the red circles (first row)
highlight the refinement of other methods still misaligned with the 2D; the
green ones (second row) highlight the penetration problem caused by their
depth inaccuracy; and the yellow circles (third row) have both above problems.

###  5.3 Comparison with the State-of-the-art

In this subsection, we compare our approach with the SOTA human mesh
refinement methods, as presented in Tab. 2. The upper segment of Tab. 2 first
represents our HMR base model CLIFFb [21] as stated previously. The bottom
segment ensures a fair comparison as all utilize ground truth 2D keypoints. By
comparing with optimization-based methods in the second segment (_e.g_.,
CLIFFr [21] and SMPLify [2] adapted by ours), our KITRO demonstrates a
significant enhancement over them (about 15% improvement in PA-MPJPE, 20%
improvement in MPJPE, 25% improvement in PVE than SOTA refinement method
[21]). Notably, our method achieves a 27.67mm PA-MPJPE joint error on the 3DPW
dataset, which is comparably close to the 26mm accuracy obtained in the
dataset’s creation process [35]. This advancement is attributed to our
explicit depth and kinematic-tree modeling, which consistently yields a
precisely refined pose fitting with 2D clues more ideally.

We also compare our visualization result with other methods as shown in Fig.
6. Based on the same initialized human pose, our approach aligns more
accurately with 2D evidence, as evident in the upright position of the left
hand in the first row, unlike the incorrect direction seen with other methods.
Additionally, while SPMLify [2] and CLIFFr [21] may correctly fit 2D evidence,
depth ambiguities often lead to penetration issues, as in the second row in
Fig. 6. Our explicit depth modeling effectively avoids such problems.

###  5.4 Analysis

In this subsection, we analyze KITRO in three aspects: its enhancement on top
of other refinement methods; its adaptability to similar parametric models
like MANO [28] in hand mesh recovery; and its overall improvement coverage
quantified by the improvement samples proportion.

Refinement on Top of Refinement. Previous subsection shows the result on top
of HMR base models (_e.g_., CLIFFb, SPIN, EFTb). Here we further examine
whether our method can get extra improvement on top of other refinement
methods. Tab. 4 shows KITRO can further enhance the refinement result from
SOTA refinement method CLIFFr, showing KITRO can make gains they are unable to
achieve.

Table 4: Our refinement results on top of CLIFFr on 3DPW.

Method | PA-MPJPE $\downarrow$ | MPJPE $\downarrow$ | PVE $\downarrow$
---|---|---|---
CLIFFb | 43.76 | 73.67 | 91.58
\+ CLIFFr | 32.04 | 55.83 | 71.95
\+ CLIFFr \+ ours | 26.21 | 46.96 | 57.53

Generalizing to Other Parametric Model. Beyond the SMPL model, our method’s
versatility is further illustrated through its application to the MANO model
[28], a commonly used parametric model for hand mesh recovery. Lacking camera
prediction in the pre-trained MANO baseline, we utilized ground truth camera
translation for these experiments and ensured the same setting for other
adapted refinement methods for fair comparison. The results shown in Tab. 5
demonstrate that our method not only refines human mesh predictions
effectively but is also suitable for other parametric models like MANO.

Table 5: Result of our method adopted to MANO model on Freihand. PA-PVE
denotes the PVE result after Procrustes aligned.

Method | PA-MPJPE $\downarrow$ | PA-PVE $\downarrow$
---|---|---
Baseline (MANO)* | 7.82 | 8.01
Baseline + SMPLify* | 4.66 | 5.07
Baseline \+ ours | 4.06 | 4.58

Comprehensive Improvement Across Joints and Samples. Our method not only
enhances average performance but also ensures equitable and widespread
effectiveness across joints and samples. As depicted in Fig. 1(b), it
effectively enhances both proximal and distal joints, while other methods may
sacrifice the proximal ones. Additionally, a significant portion of individual
samples improved after our refinement: 88% on 3DPW and 92% on Human3.6M as
shown in Sec. E of the Supplementary. This underscores our method’s
comprehensive efficacy.

##  6 Limitations

KITRO relies on the initial predicted mesh as a reference for hypothesis
selection, which can lead to deviations in cases of poor initial mesh
predictions. This is particularly evident in scenarios of person
misidentification errors 444Predicting the wrong person in an image with
multiple individuals., where the initial 3D mesh is totally mismatched with
the 2D evidence. More discussion is in Sec. F of the Supplementary. In
addition, similar to other human mesh refinement methods, KITRO relies on the
accuracy of the input 2D pose. A detailed discussion and evaluation on the
detected and noisy 2D keypoints can be found in Sec. G of the Supplementary.

##  7 Conclusion

Motivated by the inadequate depth accuracy and suboptimal proximal joint
performance in existing human mesh refinement methods, we propose KITRO. By
explicitly modeling joint depth in closed-form solution along the human
kinematic-tree, we can then use the decision tree to trace all hypotheses and
select the most likely one. KITRO demonstrates superior accuracy and
comprehensive improvement across various datasets and baseline models.

## Acknowledgement

This research / project is supported by the Ministry of Education, Singapore,
under its MOE Academic Research Fund Tier 2 (STEM RIE2025 MOE-T2EP20220-0015).

## References

  * Baerlocher and Boulic [2001] Paolo Baerlocher and Ronan Boulic.  Parametrization and range of motion of the ball-and-socket joint.  In _Deformable Avatars: IFIP TC5/WG5. 10 DEFORM’2000 Workshop November 29–30, 2000 Geneva, Switzerland and AVATARS’2000 Workshop November 30–December 1, 2000 Lausanne, Switzerland_ , pages 180–190. Springer, 2001.
  * Bogo et al. [2016] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black.  Keep it smpl: Automatic estimation of 3d human pose and shape from a single image.  In _European Conference on Computer Vision (ECCV)_ , pages 561–578. Springer, 2016.
  * Cao et al. [2017] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7291–7299, 2017.
  * Cho et al. [2023] Hanbyel Cho, Yooshin Cho, Jaesung Ahn, and Junmo Kim.  Implicit 3d human mesh recovery using consistency with pose and shape from unseen-view.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 21148–21158, 2023.
  * Choi et al. [2020] Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee.  Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose.  In _European Conference on Computer Vision (ECCV)_ , pages 769–787. Springer, 2020.
  * Choi et al. [2021] Hongsuk Choi, Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee.  Beyond static features for temporally consistent 3d human pose and shape from a video.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Dobrowolski [2015] Przemysław Dobrowolski.  Swing-twist decomposition in clifford algebra.  _arXiv preprint arXiv:1506.05481_ , 2015.
  * Grassia [1998] F Sebastian Grassia.  Practical parameterization of rotations using the exponential map.  _Journal of graphics tools_ , 3(3):29–48, 1998\.
  * Gu et al. [2023a] Kerui Gu, Rongyu Chen, and Angela Yao.  On the calibration of human pose estimation.  _arXiv preprint arXiv:2311.17105_ , 2023a.
  * Gu et al. [2023b] Kerui Gu, Zhihao Li, Shiyong Liu, Jianzhuang Liu, Songcen Xu, Youliang Yan, Michael Bi Mi, Kenji Kawaguchi, and Angela Yao.  Learning unorthogonalized matrices for rotation estimation.  _arXiv preprint arXiv:2312.00462_ , 2023b.
  * Guan et al. [2022] Shanyan Guan, Jingwei Xu, Michelle Zhang He, Yunbo Wang, Bingbing Ni, and Xiaokang Yang.  Out-of-domain human mesh reconstruction via dynamic bilevel online adaptation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 45(4):5070–5086, 2022.
  * Joo et al. [2021] Hanbyul Joo, Natalia Neverova, and Andrea Vedaldi.  Exemplar fine-tuning for 3d human model fitting towards in-the-wild 3d human pose estimation.  In _2021 International Conference on 3D Vision (3DV)_ , pages 42–52. IEEE, 2021.
  * Kanazawa et al. [2018] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Kissos et al. [2020] Imry Kissos, Lior Fritz, Matan Goldman, Omer Meir, Eduard Oks, and Mark Kliger.  Beyond weak perspective for monocular 3d human pose estimation.  In _European Conference on Computer Vision Workshops (ECCVW)_ , pages 541–554. Springer, 2020.
  * Kocabas et al. [2020] Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black.  Vibe: Video inference for human body pose and shape estimation.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Kocabas et al. [2021a] Muhammed Kocabas, Chun-Hao P Huang, Otmar Hilliges, and Michael J Black.  Pare: Part attention regressor for 3d human body estimation.  In _IEEE International Conference on Computer Vision (ICCV)_ , pages 11127–11137, 2021a.
  * Kocabas et al. [2021b] Muhammed Kocabas, Chun-Hao P Huang, Joachim Tesch, Lea Müller, Otmar Hilliges, and Michael J Black.  Spec: Seeing people in the wild with an estimated camera.  In _IEEE International Conference on Computer Vision (ICCV)_ , pages 11035–11045, 2021b.
  * Kolotouros et al. [2019] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis.  Learning to reconstruct 3d human pose and shape via model-fitting in the loop.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2019\.
  * Li et al. [2021] Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu.  Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3383–3393, 2021.
  * Li et al. [2020] S. Li, L. Ke, K. Pratama, Y.-W. Tai, C.-K. Tang, and K.-T. Cheng.  Cascaded deep monocular 3d human pose estimation with evolutionary training data.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , Seattle, WA, USA, 2020.
  * Li et al. [2022] Zhihao Li, Jianzhuang Liu, Zhensong Zhang, Songcen Xu, and Youliang Yan.  Cliff: Carrying location information in full frames into human pose and shape estimation.  In _European Conference on Computer Vision (ECCV)_ , pages 590–606. Springer, 2022.
  * Lin et al. [2021a] Kevin Lin, Lijuan Wang, and Zicheng Liu.  Mesh graphormer.  In _IEEE International Conference on Computer Vision (ICCV)_ , pages 12939–12948, 2021a.
  * Lin et al. [2021b] Kevin Lin, Lijuan Wang, and Zicheng Liu.  End-to-end human pose and mesh reconstruction with transformers.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1954–1963, 2021b.
  * Loper et al. [2015] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  SMPL: A skinned multi-person linear model.  _ACM Transactions on Graphics (TOG)_ , 34(6):248:1–248:16, 2015.
  * Moon and Lee [2020] Gyeongsik Moon and Kyoung Mu Lee.  I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image.  In _European Conference on Computer Vision (ECCV)_ , pages 752–768. Springer, 2020.
  * Moon et al. [2020] Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee.  Neuralannot: Neural annotator for 3d human mesh training sets.  _arXiv preprint arXiv:2011.11232_ , 2020.
  * Muller et al. [2021] Lea Muller, Ahmed AA Osman, Siyu Tang, Chun-Hao P Huang, and Michael J Black.  On self-contact and human pose.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9990–9999, 2021.
  * Romero et al. [2017] Javier Romero, Dimitrios Tzionas, and Michael J. Black.  Embodied hands: Modeling and capturing hands and bodies together.  _ACM Transactions on Graphics, (Proc. SIGGRAPH Asia)_ , 36(6), 2017.
  * Rong et al. [2019] Yu Rong, Ziwei Liu, Cheng Li, Kaidi Cao, and Chen Change Loy.  Delving deep into hybrid annotations for 3d human recovery in the wild.  In _IEEE International Conference on Computer Vision (ICCV)_ , pages 5340–5348, 2019.
  * Saito et al. [2020] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 84–93, 2020.
  * Sengupta et al. [2020] Akash Sengupta, Ignas Budvytis, and Roberto Cipolla.  Synthetic training for accurate 3d human pose and shape estimation in the wild.  _arXiv preprint arXiv:2009.10013_ , 2020.
  * Sun et al. [2019] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang.  Deep high-resolution representation learning for human pose estimation.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5693–5703, 2019.
  * Varol et al. [2018] G. Varol, D. Ceylan, B. Russell, J. Yang, E. Yumer, I. Laptev, and C. Schmid.  Bodynet: Volumetric inference of 3d human body shapes.  In _European Conference on Computer Vision (ECCV)_ , pages 20–38. Springer International Publishing, 2018.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 30, 2017.
  * Von Marcard et al. [2018] Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll.  Recovering accurate 3d human pose in the wild using imus and a moving camera.  In _European Conference on Computer Vision (ECCV)_ , pages 601–617, 2018.
  * Wang et al. [2020] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al.  Deep high-resolution representation learning for visual recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 43(10):3349–3364, 2020.
  * Wang et al. [2023] Wenjia Wang, Yongtao Ge, Haiyi Mei, Zhongang Cai, Qingping Sun, Yanjun Wang, Chunhua Shen, Lei Yang, and Taku Komura.  Zolly: Zoom focal length correctly for perspective-distorted human mesh reconstruction.  _arXiv preprint arXiv:2303.13796_ , 2023.
  * Xu and Takano [2021] T. Xu and W. Takano.  Graph stacked hourglass networks for 3d human pose estimation.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Xu et al. [2022] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao.  Vitpose: Simple vision transformer baselines for human pose estimation.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 35:38571–38584, 2022.
  * Yao et al. [2019] P. Yao, Z. Fang, F. Wu, Y. Feng, and J. Li.  Densebody: Directly regressing dense 3d human pose and shape from a single color image.  _arXiv preprint arXiv:1903.10153_ , 2019.

Supplementary Material

##  A Details on Swing-Twist Decomposition

Swing-twist decomposition for a rotation is a fundamental technique in
computer graphics [8, 1, 7] and was first introduced to the human mesh
recovery field by HybrIK [19]. As shown in Fig. 7, for any given joint, its
rotation $\boldsymbol{\theta}_{R}\in\mathbb{SO}(3)$ can be decomposed into a 2
degree-of-freedom (DoF) swing rotation $R_{sw}$ and a 1 DoF twist rotation
$R_{tw}$, such that $\boldsymbol{\theta}_{R}=R_{sw}R_{tw}$.

Figure 7: Illustration of the swing-twist decomposition [7].

Swing Rotation. $R_{sw}$ is to rotate the bone from the template relative pose
$\vec{t_{r}}$ to the target relative orientation $\vec{p_{r}}$. This rotation
occurs around axis $\vec{n}$, defined as:

$\vec{n}=\frac{\vec{t_{r}}\times\vec{p_{r}}}{\|\vec{t_{r}}\times\vec{p_{r}}\|},$
(20)

which is orthogonal to both $\vec{t_{r}}$ and $\vec{p_{r}}$. The rotation
magnitude, denoted as $\gamma$, is the angle subtended between $\vec{t_{r}}$
and $\vec{p_{r}}$:

$\cos\gamma=\frac{\vec{t_{r}}\cdot\vec{p_{r}}}{\|\vec{t_{r}}\|\|\vec{p_{r}}\|},\quad\sin\gamma=\frac{\|\vec{t_{r}}\times\vec{p_{r}}\|}{\|\vec{t_{r}}\|\|\vec{p_{r}}\|}$
(21)

Employing Rodrigues’ rotation formula, the swing rotation $R_{sw}$ is
expressed in closed form as:

$R_{sw}=\mathcal{I}+\sin\gamma[\vec{n}]_{\times}+(1-\cos\gamma)[\vec{n}]_{\times}^{2},$
(22)

where $\mathcal{I}$ represents the $3\times 3$ identity matrix and
$[\vec{n}]_{\times}$ denotes the skew-symmetric matrix of $\vec{n}$.

Twist Rotation. Then $R_{tw}$ is to rotate the bone around bone axis
$\vec{t_{r}}$ itself. Let $\varphi$ represent the rotation angle. According to
Rodrigues’ formula, the twist rotation is given by:

$R_{tw}=\mathcal{I}+\sin\varphi\frac{[\vec{t_{r}}]_{\times}}{||\vec{t_{r}}||}+(1-\cos\varphi)\frac{[\vec{t_{r}}]_{\times}^{2}}{||\vec{t_{r}}||^{2}},$
(23)

where $[\vec{t_{r}}]_{\times}$ is the skew-symmetric matrix of $\vec{t_{r}}$.

In our work, we focus on refining the swing rotation $R_{sw}$ while preserving
the initial twist rotation $R_{tw}$ estimates. This is motivated by the
limited variability in twist angles $\varphi$ due to human physiological
constraints, as evidenced by HybrIK’s empirical studies [19]. In contrast, the
swing rotation $R_{sw}$ exhibits a more significant range of motion,
necessitating a more detailed and accurate refinement. Thus, in our approach,
we explicitly formulate the bone directions in closed form in order to refine
the swing rotation $R_{sw}$.

##  B Proof-of-concept for Solution Selection

In this section, we conduct empirical studies to validate the assumptions
discussed in Sec. 4.3 of the main paper. As shown by the red bars in Fig. 8,
87% of bones in the initial HMR estimates are correctly identified as pointing
towards or away from the camera. When a $10^{\circ}$ margin of error is
tolerated in ambiguous cases where only 10 degrees separate two solutions, the
accuracy increases to 93% as illustrated by the green bars in Fig. 8. These
results affirm the effectiveness of the original HMR model in determining bone
direction towards or away from the camera, providing a reliable prior for the
decision tree formulation of our method.

Figure 8: Classification accuracy of bone pointing towards or away from the
camera using the HMR model. Red bars indicate 87% accuracy without tolerance,
while green bars show improved accuracy up to 93% when a $10^{\circ}$ error
margin is allowed. These results highlight the HMR model’s effectiveness in
coarse-grained directionality estimation of bones.

##  C Correctness Proof for $\boldsymbol{\theta}$ update

In this section, we validate the correctness of the pose update equation
(_i.e_., Eq. 19 in Sec. 4.3 of the main paper).

###### Proof.

Assuming the refinement from the root joint up to the parent joint $\tilde{p}$
of joint $p$ is already done, the current focus is updating rotation
${\boldsymbol{\theta}_{R}^{p}}$ for joint $p$. The objective is to verify that
updating the joint rotation from $\boldsymbol{\theta}_{R}^{p}$ to
${\boldsymbol{\theta}_{R}^{p}}^{\prime}$ correctly rotate the bone direction
of bone $(p,c)$ from $\vec{\mathbf{b}}^{(p,c)}$ to
$\vec{\mathbf{b}}^{(p,c)}_{new}$. Considering the joint rotations in
$\boldsymbol{\theta}$ are all relative to each parent joint’s coordinate
system, the bone direction in the absolute coordinate system is derived as the
product of relative rotations from the root joint to the parent joint along
the kinematic chain. Hence, the global rotation for joint $p$ in the absolute
coordinate system before refinement is:

$R_{abs}^{p}=\prod_{i\in KC(p)}\boldsymbol{\theta}_{R}^{i},$ (24)

where $\prod_{i\in KC(p)}$ denotes the matrix product of rotation matrices
from the root joint to joint $p$, with $KC(p)$ representing the kinematic
chain. Considering the template relative pose (T pose) for bone $(p,c)$
denoted as $\vec{t}_{r}^{p}$, when applied with absolute rotation from Eq. 24,
yields the absolute bone direction:

$\vec{\mathbf{b}}^{(p,c)}=R_{abs}^{p}\cdot\vec{t}^{p}_{r}=\prod_{i\in
KC(p)}\boldsymbol{\theta}_{R}^{i}\cdot\vec{t}^{p}_{r},$ (25)

As mentioned in the main paper, $R_{sw}^{(p,c)}$, computed via Rodrigues’
formula, is the rotation matrix that rotates $\vec{\mathbf{b}}^{(p,c)}$ to
$\vec{\mathbf{b}}^{(p,c)}_{new}$ in the absolute coordinate system:

$\vec{\mathbf{b}}^{(p,c)}_{new}=R_{sw}^{(p,c)}\cdot\vec{\mathbf{b}}^{(p,c)}.$
(26)

Now we verify the updated absolute rotation for joint $p$ after the refinement
of Eq. 19:

${R_{abs}^{p}}^{\prime}=\prod_{i\in
KC(\tilde{p})}\boldsymbol{\theta}_{R}^{i}\cdot{\boldsymbol{\theta}_{R}^{p}}^{\prime},$
(27)

applying this new rotation to the T pose for bone $(p,c)$ results in:

$\displaystyle{R_{abs}^{p}}^{\prime}\cdot\vec{t}^{p}_{r}=$
$\displaystyle\prod_{i\in
KC(\tilde{p})}\boldsymbol{\theta}_{R}^{i}\cdot{\boldsymbol{\theta}_{R}^{p}}^{\prime}\cdot\vec{t}^{p}_{r}$
(from Eq. 27) (28) $\displaystyle=$ $\displaystyle\prod_{i\in
KC(\tilde{p})}\boldsymbol{\theta}_{R}^{i}\cdot(\prod_{i\in
KC(\tilde{p})}\boldsymbol{\theta}_{R}^{i})^{T}$ $\displaystyle\cdot
R_{sw}^{(p,c)}\cdot\prod_{i\in
KC(p)}\boldsymbol{\theta}_{R}^{i}\cdot\vec{t}^{p}_{r}$ (Main’s Eq. 19)
$\displaystyle=$ $\displaystyle R_{sw}^{(p,c)}\cdot\prod_{i\in
KC(p)}\boldsymbol{\theta}_{R}^{i}\cdot\vec{t}^{p}_{r}$ (SymMat property)
$\displaystyle=$ $\displaystyle R_{sw}^{(p,c)}\cdot\vec{\mathbf{b}}^{(p,c)}$
(from Eq. 25) $\displaystyle=$ $\displaystyle\vec{\mathbf{b}}^{(p,c)}_{new}$
(from Eq. 26)

The ‘SymMat property’ corresponds to the property of symmetry rotation
matrices, where the transpose of a rotation matrix equals its inverse. Eq. 28
demonstrates that the updated joint rotation
${\boldsymbol{\theta}_{R}^{p}}^{\prime}$ correctly modifies
$\vec{\mathbf{b}}^{(p,c)}$ to the desired absolute rotation
$\vec{\mathbf{b}}^{(p,c)}_{new}$. ∎

##  D More Ablation Studies

In this section, we present extended ablation studies for our framework
design.

Tab. 6 details the results of eight different configurations of camera, shape,
and pose refinement. The study reveals that when refining only one of these
three factors fails to achieve effective results. This ineffectiveness can be
attributed to the importance of 2D keypoints and 3D human mesh alignment in
our approach. Without the proposed alignment mechanisms in camera and shape
refinements, pose refinements alone are also insufficient.

Table 6: Detailed ablation study for three factors on 3DPW dataset. The first
row represents the baseline HMR model, and the last row depicts our full
model. The identical results in the first two rows are because updating only
the camera does not change SMPL parameters, leading to unchanged outcomes.

Camera | Shape | Pose |  PA-MPJPE $\downarrow$ |  MPJPE $\downarrow$ |  PVE $\downarrow$
---|---|---|---|---|---
✗ | ✗ | ✗ | 43.76 | 73.67 | 91.58
✓ | ✗ | ✗ | 43.76 | 73.67 | 91.58
✗ | ✓ | ✗ | 44.31 | 69.92 | 83.26
✗ | ✗ | ✓ | 45.92 | 87.33 | 100.73
✓ | ✓ | ✗ | 44.57 | 80.03 | 95.71
✓ | ✗ | ✓ | 35.00 | 69.02 | 84.25
✗ | ✓ | ✓ | 28.53 | 46.68 | 57.76
✓ | ✓ | ✓ | 27.67 | 43.53 | 53.44

In addition, we explored varying learning rates (Fig. 9) and refinement
iteration numbers (Fig. 10) for the shape refinement model as discussed in Sec
5.2 of the main paper. The results depicted in these figures demonstrate that
both the learning rate and iteration number do not significantly impact the
performance of our method. This indicates a robustness in our approach to
variations in these parameters.

Figure 9: Error with different learning rates for shape refinement. These
three line plots (PA-MPJPE, MPJPE, and PVE) illustrate our model’s performance
stability across a range of learning rates. Figure 10: Error with the number
of fine-tuning iterations. The results indicate consistent model performance
over varying iterations for shape refinement.

##  E Improvement Distribution over Samples

In this section, we illustrate the extent of improvement in individual samples
following our refinement process. Fig. 11 and Fig. 12 displays the
distribution of performance improvements on the 3DPW and Human3.6M dataset
respectively. These visualizations demonstrate that a majority of the samples
exhibit significant improvement, again demonstrating the comprehensive
efficacy of our method.

Figure 11: Distribution of performance improvement on 3DPW. Figure 12:
Distribution of performance improvement on the Human3.6M.

(a) Our robustness to high-errors.

(b) Impact of 2D keypoint quality.

Figure 14: Illustration of gradient conflict: conflicting gradients at the
Elbow and Wrist can complicate and negatively impact the optimization of the
Shoulder joint, demonstrating a key limitation in previous gradient-based
human mesh refinement methods.

##  F Impact of Initial Predicted Mesh Quality

As mentioned in the limitation section, our method relies on the initial
predicted mesh as a reference for hypothesis selection. Here we investigate
how poor initial mesh predictions could impact the improvement. As shown in
Fig. 13(a), where we plot the MPJPE improvement w.r.t. the original MPJPE
error, KITRO withstands high errors with consistently larger improvements. The
reason is our global path selection with the decision tree tolerates a few
initial bone-facing errors. Additionally, about half of the 13% bone-facing
errors are from similar-valued solutions which are not serious mistakes.
However, when the initial prediction is too wrong, e.g. 300mm MPJPE or too
many bone-facing errors (see Tab. 7), then our improvements decrease.

Table 7: Impact of base models on bone-facing and PA-MPJPE.

Base Model | # of Correct Bone-facing | PA-MPJPE
---|---|---
(out of 23 bones) | Ours | SMPLify
CLIFFb |  20.1 $\pm$ 1.8 | 27.67 | 36.11
EFT |  19.5 $\pm$ 2.1 | 32.34 | 44.69
SPIN |  18.8 $\pm$ 2.4 | 42.46 | 47.99

##  G Impact of Input 2D Keypoint Quality

We follow the protocol of previous human mesh refinement works [21, 12, 5, 2,
11], we and all these works refine 3D pose and shape estimates with ground
truth 2D keypoints. Given the different conventions between 2D detectors and
SMPL regarding the definition of joints [24], it’s non-trivial to directly use
detected 2D keypoints. For example, SMPL puts the hip joint where the bone
rotation happens while Openpose [3] locates it at the surface landmark where
the thigh begins, so one needs to define or learn a mapping from Openpose to
SMPL format, or directly train a detector upon SMPL format. Here we performed
a rudimentary experiment to train a basic neural network architecture—a simple
3-layer MLP—on 3DPW training data to get the mapping from Openpose to SMPL
format. As shown in Tab. 8, the performance of both our method and SMPLify
diminishes when subjected to noisy 2D keypoints detection and mapping. This
degradation is expected as human mesh refinement inherently relies on the
precision of 2D keypoints. Nevertheless, our method is still more robust than
SMPLify under the detected keypoints scenario. It is important to note that
the current approach, utilizing only an MLP for keypoint format mapping,
leaves room for enhancement through more advanced mapping strategies and more
training data. However, those explorations extend beyond the scope of this
paper and are left for future investigation.

Table 8: Refinement results using 2D keypoints mapping from Openpose detection
result. ‘2DKP’ denotes ’2D Keypoints’; ‘GT’ denotes ’Ground Truth’.

Method | 2DKP |  PA-MPJPE $\downarrow$ |  MPJPE $\downarrow$ |  PVE $\downarrow$
---|---|---|---|---
SMPLify | GT | 39.99 | 71.11 | 84.28
Ours | GT | 27.45 | 48.42 | 59.65
SMPLify | Detected | 51.35 | 90.68 | 107.41
Ours | Detected | 45.88 | 79.80 | 96.60

Furthermore, we add Gaussian noise under different standard deviations to
simulate the poor 2D keypoint quality as shown in Fig. 13(b). Our raw method
inherently assumes that the 2D pose is correct and as such, can only withstand
small errors (see Fig. 13(b), 1-2 px std). This assumption does not hold for
larger errors and some intervention is required. A simple strategy is to add a
GMM prior, similar to SMPLify. SMPLify uses the GMM as a loss; we use it as a
likelihood to reject super unnaturally refined outputs based on erroneous 2D
poses (see Fig. 13(b), 10 px std). More effective filtering strategies to
improve overall robustness to 2D pose error are outside our current scope and
are left for future work.

##  H Joints and Bones Names on Kinematic-tree

According to SMPL [24], there are 24 joints and 23 bones defined in the human
kinematic-tree. We list all joint names and bones in Fig. 15

Figure 15: The definition of all 24 joints and 23 bones in the human
kinematic-tree used by our approach.

##  I More Visualization Results

In this section, we present additional visualization results. Fig. 16
illustrates the refinement process over iterations on more examples, and Fig.
17 provides further comparisons with SMPLify [2] and CLIFFr [21].

Figure 16: More iterative refinement visualizations. Figure 17: More
comparison visualizations.

##  J Extra Discussion

In this section, we provide more discussion about our method as follows

1) Why previous methods are suboptimal in proximal joints?
As discussed in the main paper, prior human refinement methods typically
utilize parametric optimization to optimize all body joints collectively
through gradient descent. However, this scheme has limitations with gradient
descent: the gradient updates at different joints can be inconsistent or even
conflicting. For instance, the optimal update directions for the Elbow and
Wrist might significantly differ, as demonstrated in Fig. 14. Such conflicts
in gradient directions, when backpropagated to proximal joints like the
Shoulder, can lead to complications in their updates, ultimately resulting in
suboptimal outcomes in proximal joints refinement (as illustrated in Fig. 1b
of the main paper).

2) What is the computation complexity of KITRO?
As discussed in Sec. 4.3 of the main paper, the computation is efficient due
to the decision tree’s design, where the calculation depends solely on the
depth, allowing nodes at each depth level to be processed in parallel. The
most time-consuming part of our method is the Adam-based shape optimization
and the total iteration number. However, Fig. 10 and Fig. 5 (in the main
paper) show that these numbers are both relatively low for decent results. In
practice, KITRO completes testing on all 35,515 samples in 3DPW test set in 15
minutes on a single NVIDIA GeForce RTX 2080 Ti GPU. For comparison, under
identical conditions, SMPLify requires 20 minutes, and CLIFFr requires extra
fine-tuning on the whole model taking more than 10 hours.