# DragPoser: Motion Reconstruction from Variable Sparse Tracking Signals
via Latent Space Optimization

Jose Luis Ponton  0000-0001-6576-4528 Universitat Politècnica de
CatalunyaBarcelonaSpain08034<EMAIL_ADDRESS>,  Eduard Pujol
0000-0001-6406-7499 Universitat Politècnica de CatalunyaBarcelonaSpain08034
<EMAIL_ADDRESS>,  Andreas Aristidou  0000-0001-7754-0791 University
of Cyprus75, KallipoleosNicosiaCyprus1678 CYENS Centre of Excellence23,
Plateia DimarchiasNicosiaCyprus1016<EMAIL_ADDRESS>,  Carlos Andujar
0000-0002-8480-4713 Universitat Politècnica de CatalunyaBarcelonaSpain08034
<EMAIL_ADDRESS>and  Nuria Pelechano  0000-0002-1437-245X Universitat
Politècnica de CatalunyaBarcelonaSpain08034<EMAIL_ADDRESS>
###### Abstract.

High-quality motion reconstruction that smoothly and accurately follows the
user’s movements can be achieved by high-end mocap systems consisting of many
sensors. However, obtaining such animation quality with fewer input devices is
gaining popularity as it brings mocap closer to the general public. The main
challenges with using fewer trackers include the loss of end-effector accuracy
in learning-based approaches, or the lack of naturalness and smoothness in IK-
based solutions. In addition, such systems are often finely tuned to a
specific number of trackers and are highly sensitive to missing data e.g., in
scenarios where a sensor is occluded or malfunctions. In response to these
challenges, we introduce DragPoser, a novel deep-learning-based motion
reconstruction system that accurately represents hard and dynamic on-the-fly
constraints, attaining real-time high end-effectors position accuracy. This is
achieved through a pose optimization process within a structured latent space.
Our system requires only one-time training on a large human motion dataset,
and then constraints can be dynamically defined as losses, while the pose is
iteratively refined by computing the gradients of these losses within the
latent space. To further enhance our approach, we incorporate a Temporal
Predictor network, which employs a Transformer architecture to directly encode
temporality within the latent space. This network ensures the pose
optimization is confined to the manifold of valid poses and also leverages
past pose data to predict temporally coherent poses. Results demonstrate that
DragPoser surpasses both IK-based and the latest data-driven methods in
achieving precise end-effector positioning, while it produces natural poses
and temporally coherent motion. In addition, our system showcases robustness
against on-the-fly constraint modifications, and exhibits exceptional
adaptability to various input configurations and changes, even in scenarios
where data for specific end-effectors are completely absent. The versatility
and efficiency of DragPoser, yielding high-quality results, not only extend
its accessibility to a diverse user base but also enhance its performance in
real-world applications.

Motion Tracking, Character Animation, Wearable Devices, Sparse Data

††submissionid: 878††copyright: none††ccs: Computing methodologies Motion
capture††ccs: Computing methodologies Motion processing††ccs: Computing
methodologies Animation††ccs: Computing methodologies Learning paradigms
Figure 1. Our approach reconstructs complete full-body poses using only a few
sensors (highlighted in green). In contrast to state-of-the-art learning-based
methods, our technique achieves animation smoothness while at the same time
incorporating precise end-effector position constraints (left). Furthermore,
it offers the flexibility to seamlessly incorporate additional real-time
constraints, enabling graceful handling of scenarios such as missing sensor
data (middle). Notably, our versatile one-for-all architecture has the
capability to adapt to arbitrary combinations of sensors (right), eliminating
the need for modifications to the network’s architecture or retraining.

##  1\. Introduction

Human motion reconstruction and editing have gained significant attention in
diverse industries such as entertainment, sports, and rehabilitation, with
applications spanning virtual reality (VR), augmented reality (AR), training,
education, simulations, and collaborative work. In professional settings,
motion reconstruction typically relies on high-quality optical motion capture
(mocap) systems or Inertial Measurement Unit (IMU)-based systems equipped with
numerous sensors. These systems, although effective, come with significant
costs and necessitate extensive calibrations and technical expertise. However,
recent trends indicate a growing demand for cost-effective and minimally
intrusive mocap systems that leverage consumer-grade hardware for motion
reconstruction. This shift in focus aims to cater to applications in the
Metaverse or home users, targeting affordability and accessibility.
Conversely, motion editing often involves the use of non-learned inverse
kinematics (IK) methods such as CCD (Kenwright, 2012) or FABRIK (Aristidou and
Lasenby, 2011). While IK enables the rapid creation of animations using end-
effectors, it frequently leads to time-incoherent or unnatural human motion
due to the inherently underdetermined nature of the problem.

With the growing availability of extensive repositories of motion capture data
and the rapid advancements in deep learning, various data-driven methodologies
have emerged to grapple with the intricacies and challenges of human motion
reconstruction. Recent studies have focused on the reconstruction of poses
from sparse input (Ponton et al., 2023; Yi et al., 2023; Winkler et al.,
2022), aiming to reduce reliance on expensive motion capture suits. These
methodologies, leveraging their learning from vast and precise mocap datasets,
reconstruct human motion that is characterized by both temporal coherence and
naturalness, overcoming the shortcomings of traditional IK (Oreshkin et al.,
2021; Agrawal et al., 2023). However, they often encounter limitations arising
from their rigid neural network architectures, restricting users to a specific
set of sparse inputs or constraints. Additionally, they are sensitive to
faulty sensors or tracking losses. Despite their capacity to reconstruct
smooth animations, they commonly face challenges in dealing with hard
constraints, often requiring the fallback to traditional IK methods for
refining final poses (Jiang et al., 2022a; Ponton et al., 2022).

This paper introduces DragPoser, an innovative deep-learning-based system
designed for motion reconstruction. Unlike conventional methods that rely on
direct full-body pose prediction from sparse inputs, our method leverages a
structured latent space and employs a pose optimization process to accurately
represent both hard and dynamic constraints. The key difference lies in our
utilization of an optimization process within the learned latent space,
enabling dynamic guidance in the pose search. The system undergoes a singular
training phase on an extensive human motion dataset. Subsequently, constraints
are dynamically defined as losses, and the pose is iteratively refined by
computing gradients within the latent space. To ensure temporal coherence and
constrain pose optimization within the valid pose manifold, we integrate a
Temporal Predictor network, which directly encodes temporality into the latent
space. This departure from traditional direct pose prediction methods enables
a more versatile and dynamic exploration of poses, providing increased
generality and improved accuracy when enforcing rigid constraints on end
effector positions.

We extensively tested and compared our method against state-of-the-art methods
in a variety of challenging motion sequences from public datasets, with
different sensor configurations. Our results show that DragPoser outperforms
both traditional IK-based methods and the newest data-driven approaches in
nearly all error metrics. DragPoser excels at providing accurate end-effector
positioning (as in traditional IK systems) but at the same time, it generates
natural poses and temporally coherent motion, a feat that data-driven methods
typically achieve only in the absence of hard constraints. DragPoser, however,
boasts another distinct advantage: its adaptability to hardware changes. We
showcase this adaptability by demonstrating DragPoser’s robust capability to
effectively handle missing input data caused by occlusions, faulty sensors, or
communication problems. Additionally, users can dynamically add constraints in
real-time to target specific types of motions. This dual adaptability not only
reinforces the system’s resilience but also enhances its performance in real-
world applications. Upon acceptance, the code will be released to facilitate
ongoing development, comparisons, and evaluations.

##  2\. Related Work

This section briefly reviews methods utilizing sparse sensor signals to
reconstruct full-body poses. We first discuss methods focusing on motion
capture from diverse sensor information, and then we focus on data-driven
methods overcoming the limitations of traditional inverse kinematics.

###  2.1. Motion Capture with Sparse Input

#### Motion Capture with IMU sensors

Recent research in human motion reconstruction has focused on utilizing a
reduced number of Inertial Measurement Units (IMUs) attached to the body. This
approach eliminates the need for external sensors or cameras, requires no
line-of-sight, and operates seamlessly in diverse environments and lighting
conditions. Early methods (von Marcard et al., 2017) used six IMUs for an
optimization-based offline pose reconstruction, while later advancements
(Huang et al., 2018; Yi et al., 2021) employed deep learning-based models for
real-time accuracy. A key challenge with IMUs is their lack of positional
data, leading to inaccuracies in global position estimation. Recent studies,
such as Jiang et al. (2022b), have introduced Transformer-based models to
address this issue, whereas Yi et al. (2022) employs physics-based approaches
to achieve more realistic motion. However, IMU methods frequently encounter
root positional drift over time (Ami-Williams et al., 2023). In efforts to
mitigate drift, researchers such as Guzov et al. (2021); Yi et al. (2023); Lee
and Joo (2024) have integrated monocular cameras with SLAM algorithms, aiming
for more precise localization.

#### Motion Capture with Egocentric Cameras

Pose estimation using egocentric cameras has garnered considerable attention
in recent years, mainly due to offering a markerless setup. However, there are
common challenges inherent to computer vision, such as occlusion and varying
lighting conditions. Addressing these challenges, Shiratori et al. (2011) and
Rhodin et al. (2016) were among the first to employ body-worn and dual fisheye
cameras for motion capture. The technique was further refined by Xu et al.
(2019), demonstrating the efficacy of a single head-mounted fisheye camera.
Due to recent advancements in deep learning models, fisheye camera-based pose
estimation enhancements were subsequently made (Jiang and Ithapu, 2021; Tome
et al., 2019; Wang et al., 2021; Tome et al., 2023; Wang et al., 2023). Zhang
et al. (2021) tackled the issue of lens distortion through automatic
calibration, whereas the works of Yuan and Kitani (2018, 2019) incorporated
physics-based controllers to achieve realistic poses. In a more recent
development, Li et al. (2023) introduces a two-stage estimation process,
presenting a method that simplifies motion capture without relying on paired
egocentric videos and human motions. Despite significant advancements in the
field, occlusions remain a persistent challenge in computer vision techniques,
often necessitating the integration of additional sensor data to ensure robust
pose predictions. Moreover, the incorporation of cameras in consumer-grade
systems raises concerns regarding user privacy, highlighting the need for
careful consideration in the design and application of these technologies.

#### Motion Capture with VR 6-DoF Sensors

The increasing accessibility of commercial VR devices has led to significant
progress in reconstructing full-body poses using 6-DoF trackers, which capture
both position and rotation. These trackers, utilizing external sensors or
computer vision, ensure accurate global information. Early work by Dittadi et
al. (2021) used a variational autoencoder (Kingma and Welling, 2013) for pose
reconstruction from three tracking points, though without global translation
estimates. Subsequent studies by Ahuja et al. (2021) and Ponton et al. (2022)
explored matching user poses to a motion dataset, similar to the approach in
Motion Matching (Clavet, 2016). Adding another tracker on the pelvis, Yang et
al. (2021) developed a model using Gated Recurrent Units (GRUs) to predict
lower-body movements with velocity data. To capture the continuous nature of
motion, Jiang et al. (2022a) and Zheng et al. (2023) employed Transformer
encoders for real-time pose estimation. Similarly, various generative AI-based
methods have been proposed, including conditional flow-based models
(Aliakbarian et al., 2022) and diffusion models (Du et al., 2023; Castillo et
al., 2023). Autoencoders have played a pivotal role in developing structured
latent spaces for pose reconstruction. In this context, Shin et al. (2023)
focused on navigating the latent space to reconstruct poses from sparse input,
while Milef et al. (2023) introduced new interpolation and pose anomaly
detection methods to avoid traversing invalid regions of the latent space.
Additionally, Ponton et al. (2023) combined a skeleton-aware autoencoder with
learned inverse kinematics for precise full-body pose reconstruction, ensuring
better accuracy in end-effectors. Other techniques involve the development of
reinforcement learning frameworks to create natural and realistic movements,
as proposed by Winkler et al. (2022) and Ye et al. (2022), with Lee et al.
(2023) extending this to include environment interactions. These techniques
have demonstrated their ability to reconstruct high-quality, smooth motion.
However, they are often constrained by the rigidity of their deep-learning
architectures, which limits their adaptability to varying numbers of sparse
inputs or constraints. Furthermore, these methods frequently face challenges
in accurately representing hard constraints, such as precise end-effector
positioning.

###  2.2. Learned Inverse Kinematics

In robotics and computer animation, IK solvers are essential for determining
the positions and orientations of intermediate joints in a kinematic chain,
given the target positions and orientations of end-effectors. Aristidou et al.
(2018) offers a comprehensive review of popular IK approaches for human motion
reconstruction. Yet, traditional IK solvers often face scalability issues for
multi-chain characters and a balance must be struck between computational
efficiency and pose naturalness (Caserman et al., 2019).

In robotics, learned IK methods utilize neural networks to solve or explore
solutions for target end-effectors. These networks range from lightweight
feed-forward networks (Bócsi et al., 2011; Duka, 2014; Csiszar et al., 2017;
Bensadoun et al., 2022) to more complex structures like generative adversarial
networks (Ren and Ben-Tzvi, 2020) and conditional normalizing flow networks
(Ames et al., 2022). While they speed up IK computations, they have only been
studied for limited kinematic chains, such as robotic arms with fewer degrees
of freedom than the human body.

In computer animation, early work by Grochow et al. (2004) and Wu et al.
(2011) introduced a data-driven IK system using Gaussian processes for
versatile pose modeling. The work by Huang et al. (2017) further enhanced the
idea of using Gaussian models with a traditional Jacobian-based solver for
real-time pose generation. With the augmented capabilities of modern deep-
learning-based architectures, Victor et al. (2021) proposes an autoencoder-
based IK solver. A common issue in deep learning-based IK methods is their
fixed network structure, which limits their use to a specific number of sparse
inputs. To address this, Oreshkin et al. (2021) implemented prototype encoding
and residual connections (ProtoRes), which enables dynamic adjustments on the
number of end-effectors. Building on this, Voleti et al. (2022) adapted the
method for SMPL models (Loper et al., 2015) and incorporated initial pose
estimation from images. Further advancements were made by Agrawal et al.
(2023), who refined the technique to ensure pose consistency across changes.
They utilized a skeletal graph structure, enabling the encoding of hard
constraints by restricting information flow at certain joints. This adaptation
significantly enhances the workflow for artists by allowing them to maintain
base poses while editing specific body parts. However, protores-based systems
still contain predefined constraints embedded within the network structure,
limiting the ability to modify these constraints dynamically. Moreover, as
both constraints and end-effectors variability are encoded as inputs, the
network requires training across all possible permutations of sparse input and
constraints, posing scalability challenges.

##  3\. Background

In this section, we provide an overview of Variational Autoencoders (VAE), the
core concept behind our Pose Autoencoder design, detailed in Section 4.3. This
design is key in creating a structured latent space, which is subsequently
utilized in the Pose Optimizer step outlined in Section 4.4.

#### Variational Autoencoders

Unlike traditional autoencoders, which generate a latent space with no
continuity guarantee, VAEs explicitly aim to construct a latent space that is
continuous, enabling the generation of data through the interpolation of
latent representations, as well as generating random data by sampling randomly
the latent space. More formally, given a set of continuous observations
$\\{\mathbf{x}^{(i)}\\}^{\scriptscriptstyle N}_{\scriptscriptstyle i=1}$ let
us assume that an observation can be reconstructed from an unobserved
continuous random variable $\mathbf{z}$, which follows a gaussian distribution
$p(\mathbf{z})=\mathcal{N}(0,\mathbf{I})$. Next, the objective is to find a
decoder $\mathcal{D}_{\boldsymbol{\theta}}$ with parameters
$\boldsymbol{\theta}$ which maximize the marginal log-likelihood as follows:

(1) $\log
p_{\boldsymbol{\theta}}(\mathbf{x}^{(i)})=\log\int_{\mathbf{z}}p_{\boldsymbol{\theta}}(\mathbf{x}^{(i)}|\mathbf{z})p(\mathbf{z})\,dz$

However, optimizing the marginal log-likelihood is not feasible due to the
intractability of the integral. Therefore, the Evidence Lower Bound (Eq. 2)
can be used to obtain a feasible lower bound of $\log
p_{\boldsymbol{\theta}}(\mathbf{x}^{(i)})$.

(2)
$-D_{KL}\left(q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x}^{(i)})\,||\,p(\mathbf{z})\right)+\mathbb{E}_{\mathbf{z}\sim
q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x}^{(i)})}\left[\log
p_{\boldsymbol{\theta}}(\mathbf{x}^{(i)}|\mathbf{z})\right]$

where $q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x}^{(i)})$ is an approximation
of the posterior $p_{\boldsymbol{\theta}}(\mathbf{z}|\mathbf{x}^{(i)})$ with
the learned parameters $\boldsymbol{\phi}$. The approximation of the posterior
$q_{\boldsymbol{\phi}}$ can be represented as a diagonal Gaussian, with means
and variance predicted by the encoder
$\mathcal{E}_{\boldsymbol{\phi}}=(\mu_{\boldsymbol{\phi}},\sigma_{\boldsymbol{\phi}})$,
as follows:

(3)
$q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x}^{(i)})=\mathcal{N}\left(\mu_{\boldsymbol{\phi}}(\mathbf{x}^{(i)}),\mathbf{I}\,\sigma_{\boldsymbol{\phi}}(\mathbf{x}^{(i)})\right)$

To optimize the parameters $(\boldsymbol{\theta},\boldsymbol{\phi})$ the goal
is to maximize Eq. 2. The first term is the KL divergence, denoted as
$\mathcal{L}_{KLD}$ in the following sections. This term encourages the
$q_{\boldsymbol{\phi}}$ distribution to closely resemble the normal
distribution $\mathcal{N}(\mathbf{0},\mathbf{I})$. Additionally, it structures
the latent space effectively, ensuring that when sampling
$\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ and passing it through the
decoder $\mathcal{D}_{\boldsymbol{\theta}}(\mathbf{z})$ a valid observation
$\mathbf{\hat{x}}$ is generated. The second term of Eq.2 is the autoencoder
reconstruction loss. We will define it later as a combination of losses
specifically crafted for our problem.

##  4\. Method

In this section, we first state the problem in Section 4.1, and then we
provide an overview of the proposed solution (refer to Section 4.2).
Subsequently, we present the architecture designed for full-body motion
reconstruction from sparse input in Section 4.3, while, in Section 4.4, we
offer a detailed explanation of how these components are employed during
inference.

###  4.1. Problem Definition

Human motion can be defined as an ordered sequence of $T$ poses
$(\mathbf{x}^{(i)})^{\scriptscriptstyle T}_{\scriptscriptstyle i=1}$. These
poses are based on a humanoid skeleton comprising $J$ joints. In contrast to
common hierarchical skeleton definitions, where joints are usually defined
within the local frame of their parent, we adopt a distinct approach. In our
model, joints are defined relative to the root space, utilizing quaternions.
In order to ensure that our model is invariant to global transformations, we
avoid encoding the root joint with explicit global information. Instead, the
root joint is expressed through a quaternion encoding the root’s rotational
increment and a 3D vector encoding displacement, both with respect to the
previous pose, making $\mathbf{x}^{(i)}\in\mathbb{R}^{J\times 4+3}$.

In our system, the input comprises an ordered sequence of $T$ sparse poses
$\mathbf{s}^{(i)}$, encompassing only a subset of joints $S\leq J$ (e.g., end-
effector data from VR controllers). Unlike complete poses, each sparse pose is
characterized by the global positions (represented as 3D vectors) and
rotations (represented as quaternions) of the sparse tracking signals,
resulting in $\mathbf{s}^{(i)}\in\mathbb{R}^{S\times(3+4)}$. The objective of
our task is to synthesize full-body human motion
$(\mathbf{x}^{(i)})^{\scriptscriptstyle T}_{\scriptscriptstyle i=1}$ from
sparse inputs $(\mathbf{s}^{(i)})^{\scriptscriptstyle T}_{\scriptscriptstyle
i=1}$.

###  4.2. Overview

This section outlines our method for synthesizing continuous full-body motion
from sparse input. DragPoser is comprised of a Pose Autoencoder and a Temporal
Predictor. These components are used in the pose optimizer step to generate
the final motion, as depicted in Figure 4. Our pose optimizer method relies on
having a structured and continuous latent space generated by the pose
autoencoder (see Figure 2). During the optimization process, we start with an
initial pose and a set of sparse tracking signals. We then conduct an
optimization-based search within the latent space to identify a feasible pose
that meets our specified constraints. This direct optimization in the latent
space enables us to accurately fulfill hard constraints and generate human-
like poses similar to those the pose autoencoder was trained on. Additionally,
a temporal predictor guides the optimization process to ensure temporal
consistency between poses.

###  4.3. Architecture

Figure 2. Pose Autoencoder architecture for creating a structured latent space
representing valid poses. Following the Variational Autoencoder approach, we
train the network, beginning with the input pose data represented in dual
quaternions to capture joint rotations and translations in root space. The
Probabilistic Encoder $\mathcal{E}$ estimates the mean and variance parameters
of the Gaussian distribution for the input pose. Subsequently, a latent vector
$\mathbf{z}$ is sampled and passed into the Probabilistic Decoder
$\mathcal{D}$ for pose reconstruction. Training involves two reconstruction
losses: $\mathcal{L}_{Q}$ and $\mathcal{L}_{FK}$, in addition to
$\mathcal{L}_{KLD}$ and $\mathcal{L}_{C}$ which structure the latent space,
ensuring standard Gaussian alignment and latent space continuity,
respectively.

#### Pose Autoencoder

Our motion synthesis method relies on an optimization process in a latent
space (explained in Section 4.4). The Pose Autoencoder component is used to
construct this latent space from motion capture data. Autoencoders rely on the
idea of having a bottleneck between two networks—Encoder and Decoder—which
allows the networks to effectively project the supplied data to a space with
fewer dimensions, known as latent space. However, this latent space is not
explicitly structured and, thus, may have regions with invalid points. To
address this, we based our Pose Autoencoder on the Variational Autoencoder
(VAE) architecture (see Section 3), which forces the latent space to represent
valid points in a specific domain. To design the Encoder and Decoder networks,
we require a network capable of accurately reconstructing a large amount of
motion capture data. We build on top of the work by Ponton et al. (2023) which
proposes an autoencoder structure, with no explicit structure on the latent
space, that uses skeleton-aware operations as building blocks (Aberman et al.,
2020). Next, we introduce all modifications needed to construct our Pose
Autoencoder; please refer to Ponton et al. (2023) for details on the skeleton-
aware operations.

#### Probabilistic Encoder $\mathcal{E}$

Our Pose Autoencoder reconstructs single poses
$\mathbf{x}^{(i)}\in\mathbb{R}^{J\times 4+3}$ as detailed in Section 4.1.
However, the Encoder $\mathcal{E}$ input is amplified with each joint’s root
space translations. This information helps the Encoder understand human motion
better and explicitly encodes information about the dimensions of the
skeleton. For this, we use dual quaternions (Andreou et al., 2022), which
provide a unified and compact representation encoding both the rotational and
translation information in orthogonal quaternions. Note that we encode the
root’s world displacement in its corresponding dual quaternion. Thus, we
define the function $DQ:\mathbb{R}^{J\times 4+3}\mapsto\mathbb{R}^{J\times
8}$, converting our original representation to dual quaternions.

Given a pose $\mathbf{x}^{(i)}$, we convert it to the dual quaternions
representation $DQ(\mathbf{x}^{(i)})$ and use the Encoder to project it to a
latent vector $\mathbf{z}^{(i)}\in\mathbb{R}^{L}$ with dimension $L$ (in our
tests $L=24$ for $J=22$). However, as explained in Section 3, we do not use
the Encoder’s output directly as the latent vector. We use two linear layers
to project the Encoder’s output to the mean and variance with $L$ dimensions.
Thus, we represent the Encoder $\mathcal{E}_{\boldsymbol{\phi}}$ with
parameters $\boldsymbol{\phi}$ as the tuple
$(\mu_{\boldsymbol{\phi}},\sigma_{\boldsymbol{\phi}})$. The final mean and
variance represent the Gaussian distribution of the input observation in the
latent space. The resulting latent vector $\mathbf{z}^{(i)}$ is computed as
follows:

(4)
$\mathbf{z}^{(i)}=\mu_{\boldsymbol{\phi}}\\!\left(DQ(\mathbf{x}^{(i)})\right)+\mathbf{I}\,\sigma_{\boldsymbol{\phi}}\\!\left(DQ(\mathbf{x}^{(i)})\right)\epsilon$

where $\epsilon\sim\mathcal{N}(0,\mathbf{I})$.

#### Probabilistic Decoder $\mathcal{D}$

Once the latent vector $\mathbf{z}^{(i)}$ is formed, we use a linear layer and
the Decoder $\mathcal{D}_{\boldsymbol{\theta}}$ with parameters
$\boldsymbol{\theta}$ to reconstruct the original pose $\mathbf{x}^{(i)}$ as
follows:

(5)
$\mathbf{\hat{x}}^{(i)}=\mathcal{D}_{\boldsymbol{\theta}}(\mathbf{z}^{(i)})$

#### Reconstruction Loss $\mathcal{L}_{Q}$

We utilize a regular Mean Squared Error reconstruction loss, defined as
follows:

(6) $\mathcal{L}_{Q}=MSE(\mathbf{x}^{(i)},\mathbf{\hat{x}}^{(i)})$

#### Forward Kinematics Loss $\mathcal{L}_{FK}$

To facilitate the neural network’s understanding of the joint hierarchy, we
additionally include an FK-based loss (Pavllo et al., 2018, 2020). We compute
the global position of each joint using FK and compare it with the ground
truth. Thus, we define the function $FK:\mathbb{R}^{J\times
4+3}\mapsto\mathbb{R}^{J\times 3}$, and the loss as follows:

(7)
$\mathcal{L}_{FK}=MSE\left(FK(\mathbf{x}^{(i)}),FK(\mathbf{\hat{x}}^{(i)})\right)$

#### KL Divergence Loss $\mathcal{L}_{KLD}$

This loss is inherent to the Variational Autoencoder architecture as
introduced in Section 3. It tries to force all the Encoder output
distributions to have a small KL divergence regarding the desired distribution
of the latent space—a Gaussian with mean zero and variance one. This allows us
to traverse the latent space and find valid poses around the desired
distribution in the optimization process. We define $\mathcal{L}_{KLD}$ as
follows:

(8) $\displaystyle\mathcal{L}_{KLD}$
$\displaystyle=\langle{1+\log\left(\sigma^{(i)}\right)-\mu^{(i)}-\sigma^{(i)},\mathbf{1}}\rangle$
(9) $\displaystyle\mu^{(i)}$
$\displaystyle=\mu_{\boldsymbol{\phi}}\\!\left(DQ(\mathbf{x}^{(i)})\right)$
(10) $\displaystyle\sigma^{(i)}$
$\displaystyle=\sigma_{\boldsymbol{\phi}}\\!\left(DQ(\mathbf{x}^{(i)})\right)$

#### Continuity Loss $\mathcal{L}_{C}$

Finally, we propose the Continuity Loss to improve the Pose Optimizer process
(see Section 4.4). Given two consecutive poses ($\mathbf{x}^{(i)}$,
$\mathbf{x}^{(i+1)}$), we want them together in the latent space so that with
one optimization step we can change from the initial pose to the next one:
$\hat{\mathbf{z}}^{(i+1)}=\mathbf{z}^{(i)}-\nabla_{\mathbf{z}}MSE(\hat{\mathbf{x}}^{(i)},\mathbf{x}^{(i+1)})$.
Although the KLD Loss $\mathcal{L}_{KLD}$ enforces similar poses to be
together in the latent space, it does not have the notion of temporal
continuity. With the continuity loss, we further guide the latent space
construction process to facilitate the Pose Optimizer process. The loss is
defined as follows:

(11)
$\displaystyle\mathcal{L}_{C}=MSE\left(\mathbf{x}^{(i+1)},\mathcal{D}_{\boldsymbol{\theta}}(\hat{\mathbf{z}}^{(i+1)})\right)$
(12)
$\displaystyle\hat{\mathbf{z}}^{(i+1)}=\mathbf{z}^{(i)}-\nabla_{\mathbf{z}}MSE(\hat{\mathbf{x}}^{(i)},\mathbf{x}^{(i+1)})$
(13)
$\displaystyle\nabla_{\mathbf{z}}MSE(\hat{\mathbf{x}}^{(i)},\mathbf{x}^{(i+1)})=\mathbf{J}^{\mathrm{T}}_{\mathbf{z}}\mathcal{D}_{\boldsymbol{\theta}}(\mathbf{z}^{(i)})\cdot
2(\mathbf{x}^{(i+1)}-\hat{\mathbf{x}}^{(i)})$

Figure 3 illustrates the role of the Continuity Loss ($\mathcal{L}_{C}$) in
structuring the latent space based on the temporal sequence of poses. In the
absence of $\mathcal{L}_{C}$, the Pose Autoencoder clusters similar poses
without considering their temporal sequence, which may result in an
undesirable pose being traversed between two consecutive poses during
optimization. This can cause the optimization process to become trapped in
local minima. Conversely, incorporating $\mathcal{L}_{C}$ encourages the
latent space to organize itself by pose similarity and temporal succession,
ensuring that one pose leads smoothly to the next. This structure
significantly reduces the risk of encountering local minima and facilitates a
more reliable pose transition within the optimization step.

Figure 3. Demonstration of how the continuity loss $\mathcal{L}_{C}$
structures the latent space to reflect the temporality of data. The left
diagram shows a latent space learned without $\mathcal{L}_{C}$, where similar
poses are clustered together, but transitioning smoothly from one pose to the
next is not guaranteed—potentially resulting in getting stuck in local minima.
The right diagram illustrates the latent space with $\mathcal{L}_{C}$,
encouraging consecutive poses from the original motion sequence to be within
reach of a single optimization step. Notice the ability to transition from the
last green pose to either the first red or blue pose, representing two
potential future sequences. Each pose is represented as a 2D Gaussian
distribution, while the mean predicted by the Encoder is represented with a
symbol.

#### Temporal Predictor

The Pose Autoencoder effectively constructs a structured latent space for
decoding single poses. However, this space lacks two critical properties.
First, it is unbounded, assuming arbitrary shapes as combinations of Gaussian
distributions. Second, while the Continuity Loss promotes poses to be
reachable during the optimization step, it remains unclear which pose to
select for temporal coherence among all possible reachable poses. This implies
a lack of a clear understanding of the sequential progression of poses during
optimization. To address these issues, we propose training a neural
network—Temporal Predictor—to predict the most likely sequences of latent
vectors based on their coherence with previous sequences. These predictions
can then inform and guide the Pose Optimizer process. This approach not only
helps navigate away from invalid regions in the latent space but also provides
the system with an understanding of the temporal sequence of the data. A
visual overview of the Temporal Predictor is depicted in Figure 4.

To maximize the span of previous and future latent vector sequences, without
significantly enlarging the network, the Temporal Predictor $\mathcal{T}$
operates at a reduced frequency. Specifically, $\mathcal{T}$ is executed every
$n$ frames, with $n=4$ in our experiments. We previously used $i$ to denote
frames; now, we introduce a new index $j=\lceil i/n\rceil$. For indexing
standard latent vectors $\mathbf{z}^{(i)}$, our frame rate is set to 60 fps,
which corresponds to the standard frame rate of the Pose Optimizer. For
indexing predicted latent vectors by the Temporal Predictor
$\mathbf{z_{t}}^{(j)}$, we use the lower frame rate. The Temporal Predictor
$\mathcal{T}_{\boldsymbol{\psi}}$, with parameters $\boldsymbol{\psi}$, adopts
the Transformer architecture, comprising Encoder
($\mathcal{T}^{E}_{\boldsymbol{\psi}}$) and Decoder
($\mathcal{T}^{D}_{\boldsymbol{\psi}}$) models.

The process begins with a current Temporal Predictor index $j$, a desired past
window $W_{p}$, and a future window $W_{f}$ for predicted latent vectors. We
define $p$ as the index for initiating new latent vector predictions.
Specifically, the Encoder $\mathcal{T}^{E}_{\boldsymbol{\psi}}$ is activated
every $W_{f}$ steps, with $p=W_{f}\lfloor j/W_{f}\rfloor$. Its input is a
sequence of past latent vectors $(\mathbf{z}^{(n\cdot k)}){\scriptscriptstyle
p-W_{p}\leq k<p}$. Subsequently, the Decoder
$\mathcal{T}^{D}_{\boldsymbol{\psi}}$, using the Encoder’s output and the
initial latent vector $\mathbf{z}^{(n\cdot p)}$, predicts the next latent
vector $\hat{\mathbf{z}}^{(j)}$. This output is then concatenated to the input
of the Decoder, and the process repeats $W_{f}$ times, similarly to the
execution of Transformers in Natural Language Processing tasks. The Decoder’s
input is formally defined as $\left(\mathbf{z}^{(n\cdot
p)},(\mathbf{z_{t}}^{(k)}){\scriptscriptstyle p<k<j}\right)$. Additionally,
root displacements and the height to the ground for the root, head, hands, and
feet are provided to the Encoder $\mathcal{T}^{E}_{\boldsymbol{\psi}}$ to
narrow down the search space.

The Encoder’s role is to contextualize the Temporal Predictor’s latent vector
prediction. The iterative prediction of latent vectors in the Decoder,
however, addresses different scenarios. For sparse inputs with minimal
ambiguity, like six sensors (one per limb and the root), we set $W_{f}=1$.
Therefore, the Temporal Predictor is only helping the Pose Optimizer process
to stay within the distribution of valid poses. In cases of higher ambiguity,
like four sensors (head, hands, and root), we increase $W_{f}$ to 16 or 60,
letting the Decoder predict the next possible latent vectors. The Encoder
handles actual past data and the Decoder makes future predictions. If the
Decoder mainly gets data from the Encoder, we leave little space for
prediction. Therefore, increasing $W_{f}$ effectively helps during the
prediction process.

###  4.4. Pose Optimizer

Figure 4. Motion reconstruction process via latent space optimization. The
procedure begins with an initial latent vector from which the pose is decoded.
Constraints are then dynamically established as loss functions, and the
gradients of these losses are computed via backpropagation within the latent
space. The decoded pose is iteratively refined through this process, either
until a maximum iteration count is reached or certain predefined metrics are
satisfied. This optimization is guided by a Temporal Predictor network that
employs a Transformer architecture. The Encoder module generates features from
past poses, and the Decoder module predicts future poses. While the Pose
Optimizer operates at every application frame, the Temporal Predictor is
executed at every $n$ frames (with $n=4$ in our experiments).

The preceding sections detailed the components for creating a structured
latent space to generate continuous and valid poses. However, using the Pose
Autoencoder, we cannot directly synthesize full-body motion from sparse input.
Taking inspiration from Pan et al. (2023), who demonstrate image modification
in a GAN’s latent space through optimization, we crafted a similar latent
space optimization process for pose prediction. This process is visually
summarized in Figure 4 and outlined in Algorithm 1.

for  _$i=0$ _to_ $T$_ do

$\hat{\mathbf{z}}^{(i)}\leftarrow\mathbf{z}^{(i-1)}$

$\mathbf{s}^{(i)}\leftarrow\text{Target Sparse Input}$

$j\leftarrow\lceil i/n\rceil$

$p\leftarrow W_{f}\lfloor j/W_{f}\rfloor$

if  _$j$ % $W_{f}$ == $0$_ then
$\mathbf{h}\leftarrow\mathcal{T}^{E}_{\boldsymbol{\psi}}\left((\mathbf{z}^{(n\cdot
k)}){\scriptscriptstyle p-W_{p}\leq k<p}\right)$

if  _$i$ % $n$ == $0$_ then
$\mathbf{z_{t}}^{(j)}\leftarrow\mathcal{T}^{D}_{\boldsymbol{\psi}}\left(\mathbf{z}^{(n\cdot
p)},(\mathbf{z_{t}}^{(k)}){\scriptscriptstyle p<k<j};\mathbf{h}\right)$

while  _$MAX\\_IT$ and $\mathcal{L}_{PO}>\epsilon$_ do

$\hat{\mathbf{x}}^{(i)}\leftarrow\mathcal{D}_{\theta}(\hat{\mathbf{z}}^{(i)})$

$\mathcal{L}_{PO}\leftarrow
MSE\left(FK^{\prime}(\mathbf{\hat{x}}^{(i)}),\mathbf{s}^{(i)}\right)$

$\hat{\mathbf{z}}^{(i)}\leftarrow\hat{\mathbf{z}}^{(i)}-\lambda_{PO}\nabla_{\mathbf{z}}\mathcal{L}_{PO}-\lambda_{T}\nabla_{\mathbf{z}}MSE(\hat{\mathbf{z}}^{(i)},\mathbf{z_{t}}^{(j)})$

end while

$\mathbf{z}^{(i)}\leftarrow\hat{\mathbf{z}}^{(i)}$

end for

ALGORITHM 1 Motion synthesis from sparse input
$(\mathbf{s}^{(i)})^{\scriptscriptstyle T}_{\scriptscriptstyle i=1}$.

Given a frame $i$, with its sparse input $\mathbf{s}^{(i)}$, and an initial
pose in latent form $\mathbf{z}^{(i-1)}$, our goal is to predict the pose
$\hat{\mathbf{x}}^{(i)}$ that best satisfies the constraints defined. We begin
by decoding the initial latent vector to a pose using the Decoder of the Pose
Autoencoder with fixed parameters:
$\hat{\mathbf{x}}^{(i-1)}=\mathcal{D}_{\boldsymbol{\theta}}(\mathbf{z}^{(i-1)})$.
We then employ the sparse Forward Kinematics function
$FK^{\prime}:\mathbb{R}^{J\times 4+3}\mapsto\mathbb{R}^{S\times(4+3)}$ to
match the full-body pose with the global joint positions of the sparse input.
The pose optimization loss,
$\mathcal{L}_{PO}=MSE\left(FK^{\prime}(\hat{\mathbf{x}}^{(i)}),\mathbf{s}^{(i)}\right)$,
encapsulates our constraints, which can be extended as needed, for instance,
by adding or removing sensors on the fly, or imposing arbitrary limits on
specific joints.

We backpropagate this loss through the Decoder to the latent space, obtaining
a gradient to guide the update of $\mathbf{z}^{(i-1)}$ towards fulfilling the
constraints. This process is similar to that of training the Decoder, but this
time the parameters are fixed. The updated latent vector,
$\hat{\mathbf{z}}^{(i)}=\mathbf{z}^{(i-1)}-\lambda_{PO}\nabla_{z}\mathcal{L}_{PO}$,
is then decoded back to a pose
$\hat{\mathbf{x}}^{(i)}=\mathcal{D}_{\boldsymbol{\theta}}(\hat{\mathbf{z}}^{(i)})$.
This iterative process is repeated until the positional and rotational error
thresholds of the sparse tracking signals are satisfied or a maximum iteration
count is reached.

Additionally, we integrate the Temporal Predictor’s output (see Section 4.3),
$\mathbf{z_{t}}^{(j)}$, to avoid invalid latent space regions and to maintain
temporal coherence, particularly when sensor input is sparse. This dual
optimization, combining pose and temporal constraints, ensures our predictions
remain both valid within the latent space and temporally consistent with the
input data.

##  5\. Experiments and Evaluation

In this section, we comprehensively evaluate DragPoser, employing both
quantitative and qualitative assessments on publicly available datasets, and
comparing its performance with state-of-the-art methods. Furthermore, we
evaluate our system across various sensor setups and constraint
configurations. Finally, we conduct an ablation study to examine the impact of
DragPoser’s core components. Unless otherwise noted, our evaluations utilize 6
degrees of freedom (6-DoF) signals, which provide both positional and
rotational information. Default end-effector placements are: hip, head, hands,
and feet for 6 signals; hip, head, hands, and right foot for 5 signals; hip,
head, and hands for 4 signals; head and hands for 3 signals.

#### Datasets

For evaluation, we trained DragPoser using the DanceDB dataset (Aristidou et
al., 2019) and evaluated with the HUMAND4D (Chatzitofis et al., 2020) and SOMA
(Ghorbani and Black, 2021) datasets, both part of the AMASS collection
(Mahmood et al., 2019). Our choice of AMASS was guided by its public
availability and its use in previous works. Additionally, for generating other
images and supplemental material, we employed the SparsePoser database (Ponton
et al., 2023). This database is particularly advantageous for VR applications,
offering diverse common motions and reducing incorrect poses through the use
of IMU-based motion capture compared to the optical systems used in AMASS
datasets.

#### Metrics

To evaluate the effectiveness of our method, we employ four distinct metrics
while ensuring the root position is aligned with the ground truth data for
accurate pose evaluation. First, we measure the overall pose quality through
three metrics: the _Positional Error_ (Pos), which calculates the mean
Euclidean distance between corresponding joints in centimeters; the
_Rotational Error_ (Rot), quantifying the mean angular difference between
rotations $R_{0}$ and $R_{1}$ using the angle of the rotation matrix
$D=R_{0}R_{1}^{T}$ and the _Velocity Error_ (Vel), which calculates the mean
velocity error across all joints in centimeters per second. Additionally, we
evaluate hard constraints by computing the mean Euclidean distance error of
the provided end-effectors, excluding the root, in centimeters.

#### Training and Implementation Details

We implemented DragPoser using PyTorch (Paszke et al., 2019) and optimized the
networks with the AdamW optimizer (Loshchilov and Hutter, 2019). The system
was trained and evaluated on a PC configured with an Intel Core i7-12700k CPU,
32GB of RAM, and an NVIDIA GeForce RTX 3090 GPU. The Pose Autoencoder was
trained with a batch size of 64 and a learning rate of $10^{-4}$ over
approximately one day; we set the latent space dimension to $L=24$ and the
number of joints to $J=22$, resulting in about 168 thousand parameters. For
the Temporal Predictor, we trained it with a batch size of 512 and a learning
rate of $10^{-3}$ for approximately thirty minutes. The Transformer
hyperparameters were set to four heads, three layers each for the encoder and
decoder, a feedforward dimension of 2048, and a feature dimension of 48,
totaling approximately 1.2 million parameters. During training, we used
$W_{p}=16$ and $W_{f}=16$ and ran the Pose Autoencoder’s Encoder to generate
the training database for the Temporal Predictor. Additionally, we empirically
found that introducing random noise to the four limbs aids in regularizing the
Temporal Predictor. Thus, in every training iteration, we introduce Gaussian
noise to the latent vector, with mean and standard deviation values for each
dimension derived from the dataset, applied to each limb with a 10%
probability. We initially experimented with adding noise to arbitrary joints
in the full skeleton, but we found that targeting individual limbs was more
effective. First we train the Pose Autoencoder with a weighted combination of
all the losses presented
$\mathcal{L}=\lambda_{Q}\mathcal{L}_{Q}+\lambda_{FK}\mathcal{L}_{FK}+\lambda_{KLD}\mathcal{L}_{KLD}+\lambda_{C}\mathcal{L}_{C}$
where $\lambda_{Q}=1$, $\lambda_{FK}=100$, $\lambda_{KLD}=0.001$ and
$\lambda_{C}=1$. Subsequently, we train the Temporal Predictor with a standard
MSE loss.

#### Performance

DragPoser can be used for both offline and real-time motion reconstruction
applications. We measured the average execution times of the different
components. Note that our code is executed in Python with no special
optimizations. On average, the Temporal Predictor forward pass takes around 7
ms per frame. The other components, which run each optimization iteration,
have the following performance: the Decoder’s forward pass averages 0.63 ms,
the constraints (losses) computation takes around 4.59 ms (mainly because of
the FK), and the backpropagation through the Decoder approximately 6.91 ms.
These three processes are computed on the CPU due to the relatively
lightweight Decoder model. For real-time applications, we typically limit the
number of iterations to 1 per frame, achieving a processing time of about 19
ms per frame. In contrast, we allow up to 10 iterations per frame for offline
pose processing, which extends the processing time to roughly a maximum of 128
ms per frame. This flexibility enables DragPoser to adapt to the performance
requirements of various application scenarios.

###  5.1. Comparison

We compare DragPoser with two state-of-the-art deep-learning-based methods,
AvatarPoser (Jiang et al., 2022a) and SparsePoser (Ponton et al., 2023), as
well as FinalIK (RootMotion, 2017), a state-of-the-art IK method for animating
full-body characters. These methods were selected due to their public
availability and their representation of different sensor configurations and
approaches. AvatarPoser utilizes a Transformer encoder to generate full-body
poses from sparse input (three or four sensors) and refines the arms with an
optimization-based IK. Notably, it excels in producing high-quality poses in
scenarios with high ambiguity and limited sensors. We extended the method,
referred to here as AvatarPoser+, by adding two additional sensors on the
feet, allowing a fair comparison in six sensor setups. In addition, we compare
with SparsePoser, which is the state-of-the-art deep-learning-based method in
terms of pose quality and end-effector accuracy for six 6-DoF sensors. Their
approach is fixed to six input sensors and uses an autoencoder and learned IK
networks for full-body pose reconstruction. Both AvatarPoser and SparsePoser
were re-trained using the DanceDB dataset following the training procedure of
their official implementations. For AvatarPoser, we incorporated DanceDB into
its pre-existing training database. Finally, considering the findings by
Ponton et al. (2023) that FinalIK exhibits superior end-effector accuracy
compared to deep learning-based methods, it is employed as a baseline for
evaluating hard constraints.

Method | Sensors | Joint Error | End-Effector Error
---|---|---|---
Pos (cm) |  Rot (deg) |  Vel (cm/s) |  Pos (cm)
AvatarPoser+ | 6 | 7.66(7.60) | 11.3(10.5) | 13.4(28.5) | 11.2(7.93)
FinalIK | 6 | 3.31(4.64) | 11.5(19.2) | 6.18(21.2) | 1.16(1.22)
SparsePoser | 6 | 2.81(2.66) | 5.78(5.85) | 10.8(15.0) | 3.43(2.53)
Ours | 6 | 2.18(2.70) | 7.66(8.71) | 10.7(15.1) | 1.02(0.73)
AvatarPoser | 4 | 9.24(10.1) | 12.4(11.7) | 16.3(31.7) | 9.62(6.51)
Ours | 4 | 7.86(13.1) | 10.6(12.4) | 20.0(41.5) | 1.12(0.72)
AvatarPoser | 3 | 11.0(10.8) | 14.3(12.8) | 20.1(39.5) | 12.3(7.51)
Ours _Head+Hands_ | 3 | 15.5(15.0) | 22.0(22.3) | 34.0(50.2) | 16.1(11.4)
Ours _Hip+Hands_ | 3 | 9.95(13.7) | 14.1(15.6) | 23.1(36.2) | 1.77(1.31)
Table 1. Accuracy of the reconstructed poses compared against state-of-the-art
methods, for diverse sensor configurations. We report the mean and standard
deviation (within parenthesis) of joint errors (position, rotation, and
velocity) as well as the end-effector positional error, all of them with
respect to ground truth data. Please refer to the text for details about
training and test datasets. When using 6 sensors (placed at the hip, head,
hands, and feet), we could compare against AvatarPoser+ (i.e., (Jiang et al.,
2022b) extended to use also the two sensors on the feet), FinalIK (RootMotion,
2017), and SparsePoser (Ponton et al., 2023). For the 4-sensor configuration
(hip, head, and hands), we compare against AvatarPoser; FinalIK is not
considered since it requires all end-effectors, and SparsePoser’s architecture
is designed for a fixed 6-sensor configuration. Concerning the 3-sensor setup,
we tested our approach with a head+hands configuration and with a hip+hands
one, and compared it against AvatarPoser.

#### Quantitative

Table 1 provides an overview of quantitative comparisons, highlighting
DragPoser’s effective balance between IK-like hard constraint representation
and high-quality pose reconstruction, an attribute commonly associated with
deep-learning methods. In these experiments, we found that DragPoser faces
limitations in high-ambiguity three-sensor scenarios, particularly when
lacking hip sensor data, resulting in performance below that of AvatarPoser.

In the six sensors setup (head, hands, hip, and feet), DragPoser surpasses
AvatarPoser+ and FinalIK in pose quality, aligning with SparsePoser’s
performance in _Joint Error_. Most importantly, DragPoser excels in _End-
Effector Error_ , outperforming both deep-learning methods and slightly
improving results compared to FinalIK. This superiority is primarily
attributed to the Pose Optimization process in the latent space, which
utilizes the Pose Autoencoder’s Decoder for achieving deep-learning-like pose
quality, while concurrently allowing for the fulfillment of hard constraints
akin to traditional IK. Despite the enhancement in _Positional Error_ ,
DragPoser encounters a slight increase in _Rotational Error_. Finally, we
observe that the _Velocity Error_ performance mirrors that of SparsePoser,
with FinalIK having the best performance but with a higher standard deviation.

We also experimented with further reducing the input signals to four sensors
placed in the head, hands and hip. This is a highly challenging scenario for
leg pose reconstruction. In this context, DragPoser exhibits a slight
advantage over AvatarPoser in positional and rotational errors, however, with
a slightly higher _Velocity Error_. It is important to note that, despite the
reduced sensor count, DragPoser continues to maintain high end-effector
accuracy (in this case, we take into account only the four sensors). However,
when only three sensors are used, removing the hip sensor, DragPoser struggles
to achieve the pose quality seen in AvatarPoser. On the other hand, when the
three sensors are placed on the hip and hands, DragPoser attains comparable
results to AvatarPoser when robust constraints are incorporated. We attribute
this limitation in our method to the absence of a dedicated component for
global position prediction. As a result, when no hip sensor is available, the
method optimizes the outcome solely based on the provided sensors and the
Temporal Predictor, which is proved to be insufficient for achieving accurate
global positioning. To address this, we dynamically introduce losses, such as
enforcing proximity of the feet to the floor and minimizing the distance
between hip and head ground projections. While this adaptability highlights
DragPoser’s capability to swiftly integrate new constraints, it also
emphasizes the ongoing challenge of achieving precise global position accuracy
without a hip sensor. In summary, our findings demonstrate that DragPoser
achieves state-of-the-art performance across diverse end-effector scenarios
and excels in pose reconstruction, particularly when high end-effector
accuracy is crucial. In addition, DragPoser proves to be the most flexible
method concerning input configuration and robustness to occlusions and sensor
malfunctions, as indicated in Table 2.

Figure 5. Visual comparison of the pose reconstruction results from
AvatarPoser+ (AP+), FinalIK (FIK), SparsePoser (SP), and DragPoser (Ours), for
seven example poses shown in the top row as ground truth (GT). All methods are
reconstructed using six 6-DOF trackers as input data. We highlighted with a
zoom-in the main errors found for each method when compared against the ground
truth. Deep-learning-based methods (AvatarPoser+ and SparsePoser) often cannot
accurately place end-effectors, while traditional IK (FinalIK) produces
unnatural poses. Our method can produce high-quality poses while achieving
high end-effector accuracy.

#### Qualitative

We present a visual comparison in Figure 5, showcasing selected frames from
different poses in the evaluation datasets. In the text, we refer to specific
poses in Figure 5 as row/column. All methods utilize six 6-DoF sensors, with
end-effectors represented by green spheres. Note that some methods, like
FinalIK and DragPoser, use the ankle joints for foot end-effectors, while
others, including AvatarPoser+ and SparsePoser, use the toe joints.

Generally, FinalIK demonstrates accurate end-effector placement but often at
the cost of losing plausible poses. _FIK/2_ and _FIK/7_ exemplify this, with
incorrectly oriented elbows and significant deviations from ground truth in
the rest of the examples. Moreover, due to its reliance on ad-hoc solutions
for natural pose generation, FinalIK struggles in less common scenarios,
failing to reconstruct poses or reach end-effectors, as seen in the _FIK/5_
and _FIK/6_.

Both AvatarPoser+ and SparsePoser encounter similar challenges. While they are
able to decode high-quality poses, there are instances, such as in the _AP+/4_
and _AP+/6_ , where AvatarPoser+ fails to accurately reconstruct poses. In
terms of end-effectors, most poses from these methods do not precisely align
with the target end-effectors. We believe this happens with traditional deep-
learning-based approaches due to the use of large networks; since high-
frequency details are lost during the forward process.

DragPoser consistently generates high-quality poses across all cases,
successfully meeting the end-effector hard constraints. Notably, it excels in
reconstructing challenging poses like push-ups _Ours/6_ and kicks _Ours/4_ ,
effectively balancing pose accuracy with the fulfillment of hard constraints.

###  5.2. Dynamic Constraints Evaluation

Sensors | DoF | Joint Error | End-Effector
---|---|---|---
Pos (cm) |  Rot (deg) |  Vel (cm/s) |  Pos (cm)
3 _Head+Hands_ | 6 | 15.5(15.0) | 22.0(22.3) | 34.0(50.2) | 16.1(11.4)
3 _Hip+Hands_ | 6 | 9.95(13.7) | 14.1(15.6) | 23.1(36.2) | 1.77(1.31)
4 _Hip+Head+Hands_ | 6 | 7.86(13.1) | 10.6(12.4) | 20.0(41.5) | 1.12(0.72)
5 _Hip+Head+Hands+RFoot_ | 6 | 4.95(9.44) | 9.25(10.8) | 15.3(32.8) | 1.17(0.77)
6 _Hip+Head+Hands+Feet; Faulty 1%_ | 6 | 6.55(11.1) | 12.0(14.8) | 21.1(59.7) | 10.2(16.0)*
6 _Hip+Head+Hands+Feet; Faulty 0.5%_ | 6 | 4.50(8.50) | 9.82(12.1) | 17.0(50.4) | 5.85(11.9)*
6 _Hip+Head+Hands+Feet_ |  3 _Pos_ | 5.81(6.71) | 25.9(26.2) | 13.2(19.6) | 1.56(1.51)
6 _Hip+Head+Hands+Feet_ | 6 | 2.18(2.70) | 7.66(8.71) | 10.7(15.1) | 1.02(0.73)
Table 2. Accuracy of our method for very diverse sparse input scenarios. Each
scenario is determined by the number of sensors (3 to 6), their placement, and
potential faulty behavior (first column), as well as the DoFs of each sensor
(3 for position-only, 6 for position+rotation; second column). We report the
joint errors and the end-effector errors, as in Table 1.

One key advantage of DragPoser is its ability to dynamically adapt
constraints, accommodating hardware changes or user-specific requirements. Our
evaluations demonstrate DragPoser in various sensor configurations, showcasing
its flexibility and the potential for customized loss functions to enhance
performance in different scenarios as detailed in Table 2. The baseline
configuration employs six 6-DoF, combining positional and rotational data to
minimize ambiguity and precisely reconstruct human poses. This setup delivers
the highest fidelity in terms of pose quality and end-effector accuracy.

However, in certain applications like motion editing or computer vision, only
positional information may be available. When we switch to a configuration
with six 3-DoF sensors, DragPoser continues to generate high-quality poses.
The _Positional Error_ remains similar to the six 6-DoF results of
AvatarPoser+, and end-effector accuracy is preserved, although, as expected,
there is a notable decrease in rotational precision.

We also explore scenarios with intermittent tracker functionality, simulating
faulty sensors by randomly disconnecting one sensor per frame with a
probability of 1% or 0.5%, reconnecting it after 100 frames. Operating at 60
frames per second, this yields a challenging and unstable input scenario.
Despite these conditions, DragPoser robustly reconstructs high-quality poses,
achieving metrics on par with those of previous work under stable sensor
conditions. It is important to note that the reported end-effector error
encompasses all six sensors; thus, due to the frequent disconnections, this
error is significantly larger compared to more stable setups where only the
tracked sensors are evaluated.

Furthermore, DragPoser’s flexibility extends to its ability to work with
varying numbers of input sensors. We demonstrate its effective performance
with five, four, and three sensors, as previously discussed in comparison with
existing work.

Finally, we emphasize that DragPoser’s constraints are not limited to the
number of sensors. For example, DragPoser could be configured to align a
_look-at_ vector with the head joint’s forward axis, facilitating precise head
orientation control. Other examples range from maintaining specified distances
between joints to enforcing spatial boundaries. Moreover, hyperparameters like
the balance between positional and rotational accuracy or the prioritization
of joint accuracy are fully customizable in real time.

###  5.3. Ablation Study

Table 3 presents the results of an ablation study focusing on two critical
elements of our proposed method. We first examine the impact of omitting the
Continuity Loss $\mathcal{L}_{C}$ during the training of the Pose Autoencoder.
We then evaluate the effectiveness of the Temporal Predictor $\mathcal{T}$ by
excluding it from the Pose Optimizer process. All metrics are computed with
the six 6-DoF configuration.

Ablation | Joint Error | End-Effector Error
---|---|---
Pos (cm) |  Rot (deg) |  Vel (cm/s) |  Pos (cm)
No $\mathcal{L}_{C}$ | 2.65(2.79) | 8.39(9.52) | 9.53(13.3) | 1.50(1.13)
No $\mathcal{T}$ | 3.05(4.39) | 10.5(12.2) | 13.1(19.0) | 1.36(1.07)
Ours | 2.18(2.70) | 7.66(8.71) | 10.7(15.1) | 1.02(0.73)
Table 3. Results of the ablation study. The first row shows the accuracy
metrics if removing the continuity loss $\mathcal{L}_{C}$ when training the
Pose Autoencoder. The second row shows the impact of preventing the Pose
Optimizer from using the output from the Temporal Predictor $\mathcal{T}$. As
in previous tables, we report the joint errors and the end-effector errors.

#### Continuity Loss

Our results indicate that the inclusion of $\mathcal{L}_{C}$ generally
enhances all performance metrics by structuring the latent space more
effectively. An exception is observed in the velocity metric; this is
attributed to the fact that when the Continuity Loss is used, the optimization
process needs fewer iterations to find poses that satisfy the constraints.
Consequently, a lower learning rate might be necessary to avoid overshooting
the target.

For a practical understanding, we calculated the average number of
optimization iterations required to meet the constraint thresholds in two
distinct animation scenarios—a push-up sequence and a dance routine—setting
the maximum number of iterations at 100 per frame. With $\mathcal{L}_{C}$, the
push-up animation required 57 iterations, and the dance 22, on average.
Without $\mathcal{L}_{C}$, these numbers increased significantly to 74 and 73
iterations, respectively.

In summary, the integration of the Continuity Loss $\mathcal{L}_{C}$ not only
improves on finding higher quality poses during the optimization search but
also streamlines the pose optimization process, evidenced by a reduced number
of iterations required. This leads to a more efficient optimization cycle,
reducing the time required to run our method.

Figure 6. Poses reconstructed using four sparse tracking signals (hip, head
and hands) with (middle row) and without (last row) temporal predictor. Ground
truth is colored in green. The Temporal Predictor guides the optimization
process to produce temporally-coherent motion even in high ambiguity
scenarios, thus producing a full walking cycle during motion (left). It also
prevents implausible poses by guiding the optimization away from invalid
latent space regions (right).

#### Temporal Predictor

The purpose of the Temporal Predictor is twofold. On the one hand, it ensures
that predicted latent vectors remain within the manifold of valid poses. On
the other hand, it learns the temporality within the latent space to guide the
optimization process in high-ambiguity scenarios. In its absence, we observe a
significant decline in all performance metrics, with _Joint Error_ notably
impacted due to its direct influence on the overall pose quality and not just
the optimization of end-effector positioning.

Visual results presented in Figure 6 further illustrate the Temporal
Predictor’s role. Here, the system receives input from only four sensors
(hips, head, and hands). With the Temporal Predictor enabled, the system can
generate a complete walking cycle; even if it might be out of sync with the
ground truth. Without the Temporal Predictor, the character’s feet are static,
resulting in an unnatural _dragging_ motion.

The effectiveness of the Temporal Predictor is even more pronounced in complex
motions like push-ups. While the optimization process alone can satisfy end-
effector constraints, the Temporal Predictor plays a crucial role in achieving
temporally coherent poses. Finally, the last column demonstrates the Temporal
Predictor’s ability to avoid invalid regions in the latent space: when
disabled, the optimization process traverses invalid regions of the pose
manifold, becomes trapped, and ultimately reconstructs an implausible pose.

##  6\. Conclusions

In conclusion, our work addresses the critical challenge of achieving high-
quality motion reconstruction with a reduced number of input devices, aiming
to make motion capture more accessible to a wider audience. Our proposed
method provides a perfect equilibrium between end-effector accuracy and pose
quality. Through a pose optimization process within a structured latent space,
our approach simultaneously achieves superior real-time end-effector position
accuracy compared to state-of-the-art IK solutions, and higher pose quality
compared to advanced learning approaches like AvatarPoser and SparsePoser. In
addition, our system requires only one-time training on a large human motion
dataset, allowing for dynamic constraint definition through losses during
runtime. Unlike previous methods limited to specific tracker configurations,
our approach can adapt to flexible tracker setups and can handle potential
data loss due to occlusions or malfunctions. In such instances, while there
may be a slight reduction in pose or end-effector accuracy for certain
configurations, the structured latent space and temporal predictor enable our
method to generate plausible and smooth animations. Overall, our approach not
only expands accessibility to motion capture but also proves to be versatile
and effective in handling real-world challenges, showcasing its potential for
practical applications.

#### Limitations

Despite the notable achievements of our motion reconstruction system, there
are certain limitations that need further consideration. Firstly, in setups
with only 3 sensors available, the lack of pelvis/root position tracking has a
negative impact on the quality of our motion reconstruction. To overcome this
limitation and improve the overall system’s performance, a promising approach
would be to integrate a global position prediction module. Additionally, while
our system is trained with various user dimensions and encodes poses using
dual quaternions, the lack of comprehensive data specifically focusing on
diverse user dimensions means that we do not explicitly validate its
performance in such scenarios. Another potential limitation arises in
instances where the pose optimization process may inadvertently move the
latent vector outside the valid pose manifold. Although the inclusion of the
Temporal Predictor effectively encodes valid regions, there remains a slim
possibility of reconstructing invalid poses. While this occurrence is
infrequent, it is essential to acknowledge and address this potential
vulnerability. Furthermore, it is important to note that computing the
gradient of the decoder can be time-consuming compared to standard forward
prediction. Despite this computational demand, the optimization process
typically operates with a few iterations, ensuring relatively fast
performance.

#### Future Work

Future research directions may involve exploring the integration of various
sensor types, such as computer vision, within the dynamic constraints of the
pose optimizer. This extension would allow us to have additional sensor data,
contributing to a more robust and versatile motion capture system. To further
enhance the capabilities of our pose optimizer, we aim to introduce additional
constraints, such as look-at functionality or end-effector pose preservation
during editing. These improvements could provide more nuanced control over
reconstructed motion, meeting diverse user needs and preferences. In the
pursuit of refining the latent space structure, we plan to investigate
alternative generative approaches. This includes exploring methods that
prevent unbounded properties and ensure there are no empty spaces between
valid regions. Such exploration could result in a more compact and well-
organized latent space, thus enhancing the system’s efficiency and
performance. Lastly, despite initially dismissing the idea of directly
encoding temporality in the latent space due to increased complexity, we
recognize it as an intriguing avenue for future research. Revisiting this
concept and devising efficient strategies to manage the expanded complexity
could potentially unlock new dimensions of temporal coherence and further
improve the realism of motion reconstructions.

###### Acknowledgements.

This work has received funding from the European Union’s Horizon 2020 research
and innovation programme under HORIZON-CL4-2022-HUMAN-01 grant agreement No
101093159 (XR4ED), and from MCIN/AEI/10.13039/501100011033/FEDER ”A way to
make Europe”, UE (PID2021-122136OB-C21). This project has also received
funding from the European Union’s Horizon 2020 Research and Innovation
Programme under Grant Agreement No 739578 and the Government of the Republic
of Cyprus through the Deputy Ministry of Research, Innovation and Digital
Policy. Jose Luis Ponton was also funded by the Spanish Ministry of
Universities (FPU21/01927).

## References

  * (1)
  * Aberman et al. (2020) Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Sorkine-Hornung, Daniel Cohen-Or, and Baoquan Chen. 2020.  Skeleton-Aware Networks for Deep Motion Retargeting.  _ACM Transactions on Graphics_ 39, 4 (Aug. 2020).  https://doi.org/10.1145/3386569.3392462
  * Agrawal et al. (2023) Dhruv Agrawal, Martin Guay, Jakob Buhmann, Dominik Borer, and Robert W. Sumner. 2023.  Pose and Skeleton-aware Neural IK for Pose and Motion Editing. In _SIGGRAPH Asia 2023 Conference Papers_ (Sydney, Australia) _(SA’23)_. Association for Computing Machinery, New York, NY, USA, Article 49, 10 pages.  https://doi.org/10.1145/3610548.3618217
  * Ahuja et al. (2021) Karan Ahuja, Eyal Ofek, Mar Gonzalez-Franco, Christian Holz, and Andrew D. Wilson. 2021.  CoolMoves: User Motion Accentuation in Virtual Reality.  _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ 5, 2, Article 52 (jun 2021), 23 pages.  https://doi.org/10.1145/3463499
  * Aliakbarian et al. (2022) Sadegh Aliakbarian, Pashmina Cameron, Federica Bogo, Andrew Fitzgibbon, and Thomas J. Cashman. 2022.  FLAG: Flow-based 3D Avatar Generation from Sparse Observations. In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, New Orleans, LA, USA, 13243–13252.  https://doi.org/10.1109/CVPR52688.2022.01290
  * Ames et al. (2022) Barrett Ames, Jeremy Morgan, and George Konidaris. 2022.  IKFlow: Generating Diverse Inverse Kinematics Solutions.  _IEEE Robotics and Automation Letters_ 7, 3 (2022), 7177–7184.  https://doi.org/10.1109/LRA.2022.3181374
  * Ami-Williams et al. (2023) Temi Ami-Williams, Christina-Georgia Serghides, and Andreas Aristidou. 2023.  Digitizing Traditional Dances Under Extreme Clothing: The Case Study of Eyo.  Available online at SSRN:.  https://doi.org/10.2139/ssrn.4622975
  * Andreou et al. (2022) Nefeli Andreou, Andreas Aristidou, and Yiorgos Chrysanthou. 2022.  Pose Representations for Deep Skeletal Animation.  _Computer Graphics Forum_ 41, 8 (Dec. 2022).  https://doi.org/10.1111/cgf.14632
  * Aristidou and Lasenby (2011) Andreas Aristidou and Joan Lasenby. 2011.  FABRIK: A Fast, Iterative Solver for the Inverse Kinematics Problem.  _Graphical Models_ 73, 5 (Sept. 2011), 243–260.  https://doi.org/10.1016/j.gmod.2011.05.003
  * Aristidou et al. (2018) Andreas Aristidou, Joan Lasenby, Yiorgos Chrysanthou, and Ariel Shamir. 2018.  Inverse Kinematics Techniques in Computer Graphics: A Survey.  _Computer Graphics Forum_ 37, 6 (2018), 35–58.  https://doi.org/10.1111/cgf.13310
  * Aristidou et al. (2019) Andreas Aristidou, Ariel Shamir, and Yiorgos Chrysanthou. 2019.  Digital Dance Ethnography: Organizing Large Dance Collections.  _J. Comput. Cult. Herit._ 12, 4, Article 29 (Nov. 2019), 27 pages.  https://doi.org/10.1145/3344383
  * Bensadoun et al. (2022) Raphael Bensadoun, Shir Gur, Nitsan Blau, Tom Shenkar, and Lior Wolf. 2022.  Neural Inverse Kinematics.  _arXiv preprint arXiv:2205.10837_ (2022).
  * Bócsi et al. (2011) Botond Bócsi, Duy Nguyen-Tuong, Lehel Csató, Bernhard Schölkopf, and Jan Peters. 2011.  Learning inverse kinematics with structured prediction. In _2011 IEEE/RSJ International Conference on Intelligent Robots and Systems_. 698–703.  https://doi.org/10.1109/IROS.2011.6094666
  * Caserman et al. (2019) Polona Caserman, Augusto Garcia-Agundez, Robert Konrad, Stefan Göbel, and Ralf Steinmetz. 2019.  Real-time body tracking in virtual reality using a Vive tracker.  _Virtual Reality_ 23, 2 (2019), 155–168.
  * Castillo et al. (2023) Angela Castillo, Maria Escobar, Guillaume Jeanneret, Albert Pumarola, Pablo Arbeláez, Ali Thabet, and Artsiom Sanakoyeu. 2023.  BoDiffusion: Diffusing Sparse Observations for Full-Body Human Motion Synthesis. In _2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)_. 4223–4233.  https://doi.org/10.1109/ICCVW60793.2023.00456
  * Chatzitofis et al. (2020) Anargyros Chatzitofis, Leonidas Saroglou, Prodromos Boutis, Petros Drakoulis, Nikolaos Zioulis, Shishir Subramanyam, Bart Kevelham, Caecilia Charbonnier, Pablo Cesar, Dimitrios Zarpalas, Stefanos Kollias, and Petros Daras. 2020.  HUMAN4D: A Human-Centric Multimodal Dataset for Motions and Immersive Media.  _IEEE Access_ 8 (2020), 176241–176262.
  * Clavet (2016) Simon Clavet. 2016.  Motion Matching and The Road to Next-Gen Animation. In _Proc. of the Game Developers Conference_ _(GDC’16)_.
  * Csiszar et al. (2017) Akos Csiszar, Jan Eilers, and Alexander Verl. 2017.  On solving the inverse kinematics problem using neural networks. In _24th International Conference on Mechatronics and Machine Vision in Practice_ _(M2VIP’17)_. 1–6.  https://doi.org/10.1109/M2VIP.2017.8211457
  * Dittadi et al. (2021) Andrea Dittadi, Sebastian Dziadzio, Darren Cosker, Ben Lundell, Thomas J Cashman, and Jamie Shotton. 2021.  Full-Body Motion from a Single Head-Mounted Device: Generating SMPL Poses from Partial Observations. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 11687–11697.
  * Du et al. (2023) Yuming Du, Robin Kips, Albert Pumarola, Sebastian Starke, Ali Thabet, and Artsiom Sanakoyeu. 2023.  Avatars Grow Legs: Generating Smooth Human Motion From Sparse Tracking Inputs With Diffusion Model. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 481–490.
  * Duka (2014) Adrian-Vasile Duka. 2014.  Neural network based inverse kinematics solution for trajectory tracking of a robotic arm.  _Procedia Technology_ 12 (2014), 20–27.
  * Ghorbani and Black (2021) Nima Ghorbani and Michael J. Black. 2021.  SOMA: Solving Optical Marker-Based MoCap Automatically. In _Proc. International Conference on Computer Vision_ _(ICCV’21)_. 11117–11126.
  * Grochow et al. (2004) Keith Grochow, Steven L. Martin, Aaron Hertzmann, and Zoran Popović. 2004.  Style-Based Inverse Kinematics.  _ACM Trans. Graph._ 23, 3 (aug 2004), 522–531.  https://doi.org/10.1145/1015706.1015755
  * Guzov et al. (2021) Vladimir Guzov, Aymen Mir, Torsten Sattler, and Gerard Pons-Moll. 2021.  Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors. In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 4316–4327.  https://doi.org/10.1109/CVPR46437.2021.00430
  * Huang et al. (2017) Jing Huang, Qi Wang, Marco Fratarcangeli, Ke Yan, and Catherine Pelachaud. 2017.  Multi-Variate Gaussian-Based Inverse Kinematics.  _Computer Graphics Forum_ 36, 8 (2017), 418–428.  https://doi.org/10.1111/cgf.13089
  * Huang et al. (2018) Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael J. Black, Otmar Hilliges, and Gerard Pons-Moll. 2018.  Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time.  _ACM Transactions on Graphics_ 37, 6 (Dec. 2018), 185:1–185:15.  https://doi.org/10.1145/3272127.3275108
  * Jiang and Ithapu (2021) Hao Jiang and Vamsi Krishna Ithapu. 2021.  Egocentric Pose Estimation from Human Vision Span. In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. 10986–10994.  https://doi.org/10.1109/ICCV48922.2021.01082
  * Jiang et al. (2022a) Jiaxi Jiang, Paul Streli, Huajian Qiu, Andreas Fender, Larissa Laich, Patrick Snape, and Christian Holz. 2022a.  AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing. In _Computer Vision – ECCV 2022_ _(Lecture Notes in Computer Science)_ , Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer Nature Switzerland, Cham, 443–460.  https://doi.org/10.1007/978-3-031-20065-6_26
  * Jiang et al. (2022b) Yifeng Jiang, Yuting Ye, Deepak Gopinath, Jungdam Won, Alexander W. Winkler, and C. Karen Liu. 2022b.  Transformer Inertial Poser: Real-time Human Motion Reconstruction from Sparse IMUs with Simultaneous Terrain Generation. In _SIGGRAPH Asia 2022 Conference Papers_ _(SA ’22)_. Association for Computing Machinery, New York, NY, USA, 1–9.  https://doi.org/10.1145/3550469.3555428
  * Kenwright (2012) Ben Kenwright. 2012.  Inverse Kinematics – Cyclic Coordinate Descent (CCD).  _Journal of Graphics Tools_ 16, 4 (Oct. 2012), 177–217.  https://doi.org/10.1080/2165347X.2013.823362
  * Kingma and Welling (2013) Diederik P. Kingma and Max Welling. 2013.  Auto-Encoding Variational Bayes.  https://doi.org/10.48550/arXiv.1312.6114 arXiv:1312.6114
  * Lee and Joo (2024) Jiye Lee and Hanbyul Joo. 2024.  Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera.  https://doi.org/10.48550/arXiv.2401.00847 arXiv:2401.00847
  * Lee et al. (2023) Sunmin Lee, Sebastian Starke, Yuting Ye, Jungdam Won, and Alexander Winkler. 2023.  QuestEnvSim: Environment-Aware Simulated Motion Tracking from Sparse Sensors. In _ACM SIGGRAPH 2023 Conference Proceedings_ _(SIGGRAPH ’23)_. Association for Computing Machinery, New York, NY, USA, 1–9.  https://doi.org/10.1145/3588432.3591504
  * Li et al. (2023) Jiaman Li, C. Karen Liu, and Jiajun Wu. 2023.  Ego-Body Pose Estimation via Ego-Head Pose Estimation. In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 17142–17151.  https://doi.org/10.1109/CVPR52729.2023.01644
  * Loper et al. (2015) Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015.  SMPL: A Skinned Multi-Person Linear Model.  _ACM Transactions on Graphics_ 34, 6 (Nov. 2015), 248:1–248:16.  https://doi.org/10.1145/2816795.2818013
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled Weight Decay Regularization. In _ICLR 2019_.  https://doi.org/10.48550/arXiv.1711.05101 arXiv:1711.05101
  * Mahmood et al. (2019) Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. 2019.  AMASS: Archive of Motion Capture as Surface Shapes. In _International Conference on Computer Vision_. 5442–5451.
  * Milef et al. (2023) Nicholas Milef, Shinjiro Sueda, and Nima Khademi Kalantari. 2023.  Variational Pose Prediction with Dynamic Sample Selection from Sparse Tracking Signals.  _Computer Graphics Forum_ 42, 2 (2023), 359–369.  https://doi.org/10.1111/cgf.14767
  * Oreshkin et al. (2021) Boris N. Oreshkin, Florent Bocquelet, Felix G. Harvey, Bay Raitt, and Dominic Laflamme. 2021.  ProtoRes: Proto-Residual Network for Pose Authoring via Learned Inverse Kinematics. In _International Conference on Learning Representations_.  https://doi.org/10.48550/arXiv.2106.01981
  * Pan et al. (2023) Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. 2023.  Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold. In _ACM SIGGRAPH 2023 Conference Proceedings_ _(SIGGRAPH ’23)_. Association for Computing Machinery, New York, NY, USA, 1–11.  https://doi.org/10.1145/3588432.3591500
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library. In _Advances in Neural Information Processing Systems_ , Vol. 32. Curran Associates, Inc.  https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html
  * Pavllo et al. (2020) Dario Pavllo, Christoph Feichtenhofer, Michael Auli, and David Grangier. 2020.  Modeling Human Motion with Quaternion-Based Neural Networks.  _International Journal of Computer Vision_ 128, 4 (April 2020), 855–872.  https://doi.org/10.1007/s11263-019-01245-6
  * Pavllo et al. (2018) Dario Pavllo, David Grangier, and Michael Auli. 2018.  QuaterNet: A Quaternion-based Recurrent Model for Human Motion. In _British Machine Vision Conference (BMVC)_.
  * Ponton et al. (2022) Jose Luis Ponton, Haoran Yun, Carlos Andujar, and Nuria Pelechano. 2022.  Combining Motion Matching and Orientation Prediction to Animate Avatars for Consumer-Grade VR Devices.  _Computer Graphics Forum_ (2022).  https://doi.org/10.1111/cgf.14628
  * Ponton et al. (2023) Jose Luis Ponton, Haoran Yun, Andreas Aristidou, Carlos Andujar, and Nuria Pelechano. 2023.  SparsePoser: Real-Time Full-Body Motion Reconstruction from Sparse Data.  _ACM Trans. Graph._ 43, 1, Article 5 (Oct. 2023), 14 pages.  https://doi.org/10.1145/3625264
  * Ren and Ben-Tzvi (2020) Hailin Ren and Pinhas Ben-Tzvi. 2020.  Learning inverse kinematics and dynamics of a robotic manipulator using generative adversarial networks.  _Robotics and Autonomous Systems_ 124 (2020), 103386.  https://doi.org/10.1016/j.robot.2019.103386
  * Rhodin et al. (2016) Helge Rhodin, Christian Richardt, Dan Casas, Eldar Insafutdinov, Mohammad Shafiei, Hans-Peter Seidel, Bernt Schiele, and Christian Theobalt. 2016.  EgoCap: Egocentric Marker-Less Motion Capture with Two Fisheye Cameras.  _ACM Transactions on Graphics_ 35, 6 (Dec. 2016), 162:1–162:11.  https://doi.org/10.1145/2980179.2980235
  * RootMotion (2017) RootMotion. 2017.  _Final IK_.  http://root-motion.com/
  * Shin et al. (2023) Myungjin Shin, Dohae Lee, and In-Kwon Lee. 2023.  Utilizing Task-Generic Motion Prior to Recover Full-Body Motion from Very Sparse Signals.  https://doi.org/10.48550/arXiv.2308.15839 arXiv:2308.15839
  * Shiratori et al. (2011) Takaaki Shiratori, Hyun Soo Park, Leonid Sigal, Yaser Sheikh, and Jessica K. Hodgins. 2011.  Motion Capture from Body-Mounted Cameras.  _ACM Transactions on Graphics_ 30, 4 (July 2011), 1–10.  https://doi.org/10.1145/2010324.1964926
  * Tome et al. (2023) Denis Tome, Thiemo Alldieck, Patrick Peluse, Gerard Pons-Moll, Lourdes Agapito, Hernan Badino, and Fernando de la Torre. 2023.  SelfPose: 3D Egocentric Pose Estimation From a Headset Mounted Camera.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 45, 6 (June 2023), 6794–6806.  https://doi.org/10.1109/TPAMI.2020.3029700
  * Tome et al. (2019) Denis Tome, Patrick Peluse, Lourdes Agapito, and Hernan Badino. 2019.  xR-EgoPose: Egocentric 3D Human Pose From an HMD Camera. In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_. 7727–7737.  https://doi.org/10.1109/ICCV.2019.00782
  * Victor et al. (2021) Léon Victor, Alexandre Meyer, and Saïda Bouakaz. 2021.  Learning-based pose edition for efficient and interactive design.  _Computer Animation and Virtual Worlds_ 32, 3-4 (2021), e2013.  https://doi.org/10.1002/cav.2013
  * Voleti et al. (2022) Vikram Voleti, Boris Oreshkin, Florent Bocquelet, Félix Harvey, Louis-Simon Ménard, and Christopher Pal. 2022.  SMPL-IK: Learned Morphology-Aware Inverse Kinematics for AI Driven Artistic Workflows. In _SIGGRAPH Asia 2022 Technical Communications_ _(SA ’22)_. Association for Computing Machinery, New York, NY, USA, 1–7.  https://doi.org/10.1145/3550340.3564227
  * von Marcard et al. (2017) Timo von Marcard, Bodo Rosenhahn, Michael J. Black, and Gerard Pons-Moll. 2017.  Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs.  _Comput. Graph. Forum_ 36, 2 (may 2017), 349–360.  https://doi.org/10.1111/cgf.13131
  * Wang et al. (2021) Jian Wang, Lingjie Liu, Weipeng Xu, Kripasindhu Sarkar, and Christian Theobalt. 2021.  Estimating Egocentric 3D Human Pose in Global Space. In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. 11480–11489.  https://doi.org/10.1109/ICCV48922.2021.01130
  * Wang et al. (2023) Jian Wang, Diogo Luvizon, Weipeng Xu, Lingjie Liu, Kripasindhu Sarkar, and Christian Theobalt. 2023.  Scene-Aware Egocentric 3D Human Pose Estimation. In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 13031–13040.  https://doi.org/10.1109/CVPR52729.2023.01252
  * Winkler et al. (2022) Alexander Winkler, Jungdam Won, and Yuting Ye. 2022.  QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars. In _SIGGRAPH Asia 2022 Conference Papers_. ACM, Daegu Republic of Korea, 1–8.  https://doi.org/10.1145/3550469.3555411
  * Wu et al. (2011) Xiaomao Wu, Maxime Tournier, and Lionel Reveret. 2011.  Natural Character Posing from a Large Motion Database.  _IEEE Computer Graphics and Applications_ 31, 3 (2011), 69–77.  https://doi.org/10.1109/MCG.2009.111
  * Xu et al. (2019) Weipeng Xu, Avishek Chatterjee, Michael Zollhöfer, Helge Rhodin, Pascal Fua, Hans-Peter Seidel, and Christian Theobalt. 2019.  Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera.  _IEEE Transactions on Visualization and Computer Graphics_ 25, 5 (May 2019), 2093–2101.  https://doi.org/10.1109/TVCG.2019.2898650
  * Yang et al. (2021) Dongseok Yang, Doyeon Kim, and Sung-Hee Lee. 2021.  Lobstr: Real-time lower-body pose prediction from sparse upper-body tracking signals.  40, 2 (2021), 265–275.
  * Ye et al. (2022) Yongjing Ye, Libin Liu, Lei Hu, and Shihong Xia. 2022.  Neural3Points: Learning to Generate Physically Realistic Full-body Motion for Virtual Reality Users.  _Computer Graphics Forum_ 41, 8 (2022), 183–194.  https://doi.org/10.1111/cgf.14634
  * Yi et al. (2023) Xinyu Yi, Yuxiao Zhou, Marc Habermann, Vladislav Golyanik, Shaohua Pan, Christian Theobalt, and Feng Xu. 2023.  EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensors.  _ACM Transactions on Graphics_ 42, 4 (July 2023), 76:1–76:17.  https://doi.org/10.1145/3592099
  * Yi et al. (2022) Xinyu Yi, Yuxiao Zhou, Marc Habermann, Soshi Shimada, Vladislav Golyanik, Christian Theobalt, and Feng Xu. 2022.  Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors. In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, New Orleans, LA, USA, 13157–13168.  https://doi.org/10.1109/CVPR52688.2022.01282
  * Yi et al. (2021) Xinyu Yi, Yuxiao Zhou, and Feng Xu. 2021.  TransPose: Real-Time 3D Human Translation and Pose Estimation with Six Inertial Sensors.  _ACM Transactions on Graphics_ 40, 4 (July 2021), 86:1–86:13.  https://doi.org/10.1145/3450626.3459786
  * Yuan and Kitani (2018) Ye Yuan and Kris Kitani. 2018.  3D Ego-Pose Estimation via Imitation Learning. In _Computer Vision – ECCV 2018_ _(Lecture Notes in Computer Science)_ , Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer International Publishing, Cham, 763–778.  https://doi.org/10.1007/978-3-030-01270-0_45
  * Yuan and Kitani (2019) Ye Yuan and Kris Kitani. 2019.  Ego-Pose Estimation and Forecasting As Real-Time PD Control. In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_. 10081–10091.  https://doi.org/10.1109/ICCV.2019.01018
  * Zhang et al. (2021) Yahui Zhang, Shaodi You, and Theo Gevers. 2021.  Automatic Calibration of the Fisheye Camera for Egocentric 3D Human Pose Estimation from a Single Image. In _2021 IEEE Winter Conference on Applications of Computer Vision_ _(WACV)_. IEEE, Waikoloa, HI, USA, 1771–1780.  https://doi.org/10.1109/WACV48630.2021.00181
  * Zheng et al. (2023) Xiaozheng Zheng, Zhuo Su, Chao Wen, Zhou Xue, and Xiaojie Jin. 2023.  Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modeling. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 14678–14688.