# Degrees of Freedom Matter:
Inferring Dynamics from Point Trajectories

Yan Zhang†2, Sergey Prokudin1,3, Marko Mihajlovic1, Qianli Ma†4, Siyu Tang1
1ETH Zürich, Switzerland , 2Meshcapade
3ROCS, University Hospital Balgrist, University of Zürich, 4Nvidia

###### Abstract

Understanding the dynamics of generic 3D scenes is fundamentally challenging
in computer vision, essential in enhancing applications related to scene
reconstruction, motion tracking, and avatar creation. In this work, we address
the task as the problem of inferring dense, long-range motion of 3D points. By
observing a set of point trajectories, we aim to learn an implicit motion
field parameterized by a neural network to predict the movement of novel
points within the same domain, without relying on any data-driven or scene-
specific priors. To achieve this, our approach builds upon the recently
introduced dynamic point field model [49] that learns smooth deformation
fields between the canonical frame and individual observation frames. However,
temporal consistency between consecutive frames is neglected, and the number
of required parameters increases linearly with the sequence length due to per-
frame modeling. To address these shortcomings, we exploit the intrinsic
regularization provided by SIREN [54], and modify the input layer to produce a
spatiotemporally smooth motion field. Additionally, we analyze the motion
field Jacobian matrix, and discover that the motion degrees of freedom (DOFs)
in an infinitesimal area around a point and the network hidden variables have
different behaviors to affect the model’s representational power. This enables
us to improve the model representation capability while retaining the model
compactness. Furthermore, to reduce the risk of overfitting, we introduce a
regularization term based on the assumption of piece-wise motion smoothness.
Our experiments assess the model’s performance in predicting unseen point
trajectories and its application in temporal mesh alignment with guidance. The
results demonstrate its superiority and effectiveness. The code and data for
the project are publicly available111https://yz-
cnsdqz.github.io/eigenmotion/DOMA/.

††$\dagger$ This work was done when YZ and QM were at ETH Zürich.

Figure 1: We introduce DOMA, a compact implicit motion model designed to
capture generic dynamics of 3D scenes. By processing a 3D point $\bm{x}$ in
the canonical frame alongside a 1D time step $t$, DOMA predicts an affine
mapping, parameterized by a linear map ${\bm{A}}_{\theta}$ and a translation
vector ${\bm{u}}_{\theta}$. By leveraging the inherent regularity of the
utilized SIREN framework [55], DOMA ensures the generation of a
spatiotemporally smooth motion field. The model’s capacity to represent
complex dynamics can be controlled by adjusting the degrees of freedom of the
output affine mapping.

##  1 Introduction

Motion estimation plays a crucial role in several key areas of computer
vision, such as dynamic scene reconstruction, autonomous navigation, and
avatar creation. Treated as a distinct task, it emerges in various contexts as
non-rigid tracking [11], point set [58, 40] and mesh registration [3, 17],
shape matching [43], as well as optical and scene flow estimation [69]. The
solutions adopted in these contexts significantly differ based on the ultimate
objective and the foundational assumptions about the scene. A substantial
amount of research exists concentrating on human-centric [48, 4] and rigid
object motion [2, 64], alongside efforts in learning generic 2D motion priors
in a data-driven manner [12, 23]. The diversity in applications and approaches
underscores the complexity and significance of motion estimation within the
field of computer vision [35].

In this work, we aim to develop a motion model capable of reconstructing the
dynamics of generic 3D scenes without relying on data-driven motion priors or
object-specific models. Specifically, by analyzing observed point trajectories
within dynamic 3D scenes, we seek to learn an implicit motion model capable of
predicting the movement of novel 3D points. This task bears significant
relevance to the process of warping 3D points across frames, a procedure
commonly encountered in neural rendering [45, 30], point cloud alignment [27],
object tracking [60], and avatar creation [62, 44, 51]. Typically, warping
methods developed in these domains are intended to supplement the primary
objectives such as the quality of novel view synthesis. Consequently, critical
features such as the representational capability of the motion model, along
with the consistency and plausibility of the motion it recovers, have not been
the primary concern in these studies.

In this context, the closest work is the recently proposed dynamic point field
(DPF) model [49], which addresses the task of recovering the correct implicit
deformation function based on the observed pair of 3D surfaces. It proposes a
lightweight deformation field formulated by a SIREN network [54], an MLP with
periodic activation functions. Due to the regularity introduced by SIREN, its
modeled deformation is spatially smooth, enabling various applications such as
robust mesh registration and avatar animation.

Despite its effectiveness, the DPF method is limited to learning deformation
fields between just two frames, the canonical and the target frames. To extend
this to multiple frames, it proposes creating a set of deformation field
models, where each model transforms points from the canonical frame to a
distinct target frame. Consequently, the number of models increases with the
sequence length, leading to significant memory and computational overhead.
Moreover, the frame-wise approach fails to ensure temporal motion consistency,
potentially resulting in discontinuities between consecutive frames and
jittering artifacts. We conduct a thorough analysis of the representational
capabilities of DPF and its underlying SIREN network to address these
shortcomings. To achieve temporal smoothness, we follow the wave equation
formulation in [54], and modify the input layer by incorporating a 1D time
step along with the existing 3D point location input, similar to the strategy
regularly implied in the implicit warping fields for neural rendering [50].

Compared to the per-frame modeling scheme in DPF, this operation attempts to
compress the entire sequence into a single network, which raises challenges on
the model’s capacity of motion representation. Rather than increasing the
number of hidden variables in the SIREN network, we opt to refine its output
layer by introducing more motion degrees of freedom (DOFs). From a
mathematical standpoint of continuum mechanics [56], the advantages of
additional DOFs are demonstrated in the Jacobian matrix of the motion field:
Two points that are infinitely close to each other in space gain greater
movement flexibility, provided the same number of network hidden variables.
Therefore, the model representation power is improved, and the model
compactness is retained.

Nevertheless, additional DOFs can increase the risk of overfitting, in
particular when the observed point trajectories are excessively sparse. To
overcome this issue, we leverage a generic assumption on motion, _i.e_. piece-
wise smoothness, and propose a motion smoothness term by penalizing the
approximate L1 norm of spatial derivatives of the predicted transformations.
Here, rather than employing an auto-differentiation framework, we derive
analytical gradients of our employed SIREN network to speed up the
computation.

We undertake comprehensive experiments to validate the efficacy of our method.
To assess its motion representation capabilities, we extract seven challenging
sequences from the DeformingThings4D dataset [28] and generate four synthetic
sequences that exhibit basic 3D motions. Our method demonstrates consistently
superior performance in predicting the motion of novel points, when compared
to both state-of-the-art methods and their variations. Furthermore, we employ
our technique in the task of temporal mesh alignment with guidance, and
evaluate its performance on complex sequences from the Resynth dataset [33,
32]. Compared to the DPF baseline, our approach achieves comparable alignment
accuracy, better temporal regularity, and significantly smaller models,
occupying approximately 200KB versus 8MB in the saved checkpoint for a
30-frame sequence.

We refer to our approach as DOMA, an acronym for Degrees Of freedoM mAtter,
contending that additional degrees-of-freedom is essential to improve the
expressivity of implicit motion models. Technical contributions are summarized
as follows:

  * •

We extend the state-of-the-art implicit model for surface deformation [49] for
continuous, multi-frame motion modeling, leading to an implicit,
spatiotemporally smooth affinity field;

  * •

We leverage the Jacobian matrix to analyze the motion field complexity, and
discover that additional DOFs at the output layer improve the model
representation capability while retaining the model compactness;

  * •

To enhance the quality of the motion learned, we introduce a regularization
term based on the piece-wise smoothness assumptions of the domain;

  * •

We assess our model, demonstrating the benefits of various modeling decisions
through experiments, on challenging long-term scene flow estimation and guided
mesh alignment.

##  2 Related Work

Motion representation with object models. Given the motion of a set of points,
it is a highly unconstrained problem to infer the motion of other points in
their proximity. In many applications, such an inverse problem is solved based
on an object model that performs as a strong prior of the dynamics. Typical
examples are marker-based human motion estimation [34, 31, 71], or 3D pose
estimation from imagery data [47, 52, 21, 70], in which human parametric body
models such as [47, 66] are leveraged. Here, the bones of a body model serve
as an intermediate proxy for all other points’ motion: the trajectory of a
point on the body surface is generated by the weighted average of the bone
transformations, a technique referred to as linear blend skinning (LBS). When
extending the skinning weights to a vector field, as in [53, 36, 10, 61, 62,
65, 37], any point in the space can be animated by the bone transformations.
The same technology can be applied to animals [75], babies [18], humanoids
[68], and other categories.

In cases where the object model is not directly available, it can be jointly
optimized together with the motion from observations. This provides
flexibility on the object categories to handle more generic dynamics. For
example, BANMO [67] proposes a generic deformable model, where a set of 3D
Gaussians serve as the motion control proxy, analogous to bones. During
optimization, the Gaussian locations and orientations are optimized together
with their transformations. Likewise, KeyTr [42] proposes a bone basis to
deform a point cloud across frames, in which the basis coefficients play a
similar role as the skinning weights.

Motion representation without object models. Another line of work models the
motion of points without the reliance of intermediate proxies like bones.
Instead, they represent the motion of all points in space as a dense field, in
which each location stores a transformation matrix. The motion of a point will
be determined by the transformations of its infinitesimal neighbourhood.
Methods under this paradigm are frequently employed in neural rendering [57,
45, 30], dense tracking [60], surface reconstruction [41, 44] and non-rigid
geometry alignment [49, 27]. Niemeyer _et al_. [41] employ neural ODE [9] to
model the dynamics, and estimate the implicit occupancy function at the
canonical frame and its evolution as time progresses. Prokudin _et al_. [49],
Pumarola _et al_. [50], and Palafox _et al_. [44] leverage MLP to model a
translation field (or scene flow field), and warp the point from the canonical
frame to the target frame via addition. Li _et al_. [26] employ a neural
network to parameterize the flow field for regularization. The exploited
network is a MLP with ReLU [25] activation functions. Park _et al_. [45]
design a SE(3) transformation field, warping the points on the camera ray from
the observation frame to the canonical frame, so as to train the neural
radiance field [39] reliably. Lombardi _et al_. [30] employs a mixture of
scaled SE(3) warping fields for the purpose of neural rendering dynamic
scenes. Likewise, Li and Harada [27] apply SE(3) or scaled SE(3)
transformations to perform non-rigid point cloud alignment. Compared to the
SE(3) transformation, the scaled-SE(3) transformation is capable of
representing the dilation or shrinking of an object. Going beyond points’
locations, the spatial transformations can also be applied to features in
neural networks and potentially improve the performance on _e.g_. image
classification [20].

Relations to object shape and view recovery from images. Existing works such
as [22, 16, 15] study to learn neural models from an image collection, and
recover the 3D shape in a canonical frame, the camera pose, and the texture of
an object from a single image. Despite addressing different tasks, their
solutions of composing the instance-level shape by the mean shape and
deformation is relevant to our manner of motion modeling. Furthermore, we are
encouraged by these works to reconstruct dynamic scenes from multiview videos
as future work.

DOMA in context. Existing motion modeling approaches are developed together
with individual applications, in which the network architectures, coordinate
encodings, and other properties are diverse. The motion representation
capabilities of their models are seldom investigated. In contrast, we start
with the basic assumption that the motion field has spatiotemporal regularity.
Therefore, we leverage the SIREN [55] network, and extend the start-of-the-art
work DPF [49] to a multi-frame, smooth affinity field model. We leverage
knowledge of continuum mechanics, exploit the Jacobian matrix to describe the
motion field complexity, and find that DOFs at the output layer and the
network hidden variables affect the model representation power in different
manners. Guided by these insights, we propose a solution to increase the model
capacity while retaining the model lightweight. Moreover, we introduce a
smoothness regularization term to overcome overfitting, which does not assume
the underlying motion is _e.g_. rigid like in [45]. The effectiveness of DOMA
is demonstrated with experiments in Sec. 4.

##  3 Method

###  3.1 Preliminaries

####  3.1.1 SIREN [54]

SIREN proposes an implicit neural representation, which is a multilayer
perceptron (MLP) with periodic activation functions. Specifically, the MLP
with $n+1$ layers is given by

${\bm{y}}={\bm{W}}_{n}\left(\phi_{n-1}\circ\phi_{n-2}\circ\cdots\circ\phi_{0}\right)({\bm{x}})+{\bm{b}}_{n},$
(1)

with $\phi_{i}=\sin({\bm{W}}_{i}{\bm{x}}_{i}+{\bm{b}}_{i})$ and
$i=\\{0,1,...,n-1\\}$. The gradient of the model w.r.t. the input is another
phase-shifted SIREN network, and hence is infinitely differentiable. As
reported in [54], the sinusoidal activation functions boost the model
performance on the convergence speed, reconstruction quality, and smoothness,
letting the MLP outperform baselines consistently and considerably. However,
this network requires special initialization to be trainable. Given
${\bm{x}}\in\mathbb{R}^{d}$, it is suggested to have the weights $w_{i}$ in
the uniform distribution $\mathcal{U}(-\sqrt{6/d},\sqrt{6/d})$, so that the
model output will converge to a normal distribution.

####  3.1.2 Dynamic Point Field (DPF) [49]

DPF proposes an implicit deformation field to model point dynamics, achieving
state-of-the-art performance on surface reconstruction, geometry deformation,
and avatar animation with challenging clothing. Given a point
${\bm{x}}\in\mathbb{R}^{3}$ in the canonical frame, it learns a field
${\bm{u}}:\mathbb{R}^{3}\to\mathbb{R}^{3}$ formulated by a SIREN network [55],
and then transforms the point to a new location ${\bm{y}}$, _i.e_.

${\bm{y}}=g({\bm{x}})={\bm{x}}+{\bm{u}}({\bm{x}}).$ (2)

To handle complex and rapid motion, an as-isometric-as-possible (AIAP) loss
term [49, Eq.13] is proposed to minimize changes of pair-wise distances
between neighbor points during deformation. Furthermore, it proposes guided
geometry deformation via corresponding keypoints, which can avoid sub-optimal
matching caused by the Chamfer distance [14]. Provided a set of keypoint pairs
$\\{({\bm{v}}_{i}^{c},{\bm{v}}_{i}^{t})\\}_{i=1}^{N}$, and a pair of non-
corresponding geometries (_e.g_. meshes and point clouds) to align
$({\bm{M}}_{c},{\bm{M}}_{t})$, the guided geometry deformation can be
performed by minimizing

$\alpha_{1}\mathcal{L}_{CD}\left({\bm{M}}_{t},{\bm{g}}({\bm{x}})\right)+\alpha_{2}\mathcal{L}_{V}({\bm{v}}^{c},{\bm{v}}^{t})+\alpha_{3}\mathcal{L}_{AIAP}\left({\bm{g}}({\bm{x}}),{\bm{x}}\right),$
(3)

in which ${\bm{x}}\in{\bm{M}}_{c}$, $\alpha$s denote the loss weights, and
$\mathcal{L}_{CD}$, $\mathcal{L}_{V}$, and $\mathcal{L}_{AIAP}$ denote the
Chamfer loss, the L1 loss, and the AIAP loss on the corresponding keypoints,
respectively. To align a sequence of geometries, DPF [49] suggests learning a
set of deformation fields that warp points in the canonical frame to
individual target frames.

###  3.2 DOMA: Spatiotemporal Affinity Motion Fields

DOMA is an implicit motion field formulated by

${\bm{y}}={\bm{A}}({\bm{x}},t){\bm{x}}+{\bm{u}}({\bm{x}},t),$ (4)

in which ${\bm{x}}\in\mathbb{R}^{3}$ is a point in the canonical frame,
$t\in\mathbb{R}$ is the time step,
${\bm{A}}:\mathbb{R}^{3}\times\mathbb{R}\to\mathbb{R}^{3\times 3}$ and
${\bm{u}}:\mathbb{R}^{3}\times\mathbb{R}\to\mathbb{R}^{3}$ are formulated by a
shared SIREN network. Following Sitzmann et al. [54, Sec.5.4 of supp. mat.]
that how the wave equation is formulated and solved, we incorporate the 1D
time into SIREN as input, letting $\frac{\partial{\bm{y}}}{\partial t}$ be
another phase-shifted SIREN and get regularized.

####  3.2.1 On The Representation Power

Different from the per-frame modeling mechanism of DPF [49], incorporating the
1D time domain into the input layer compresses the entire sequence into a
single network, raising challenges on the model representation power.

Referring to [56], the DPF formula Eq. (2) is generic to model object
deformation in continuum mechanics. However, it has limitations in empirical
studies, motivating us to investigate the reasons. Rather than studying the
entire domain, we look into an infinitesimal region around an arbitrary 3D
point ${\bm{x}}$ in the canonical frame, and derive its Jacobian matrix as

$\frac{\partial{\bm{y}}}{\partial{\bm{x}}}={\bm{I}}+\nabla{\bm{u}}({\bm{x}}),$
(5)

which is the optimal linear approximation of the motion around ${\bm{x}}$ and
$\nabla$ denotes the spatial gradient.

This Jacobian matrix is able to reflect the motion complexity. Intuitively,
$\frac{\partial{\bm{y}}}{\partial{\bm{x}}}$ indicates the difference of
movements between two points that are infinitely close to each other. Without
any constraints on $\nabla{\bm{u}}$, the model is capable of representing
highly complex motion. However, ${\bm{u}}$ is formulated by SIREN [54],
letting $\nabla{\bm{u}}$ become to

$\nabla{\bm{u}}={\bm{W}}_{n}\left(\prod_{i=0}^{n-1}{\bm{W}}_{i}\circ\varphi_{i}({\bm{x}})\right),$
(6)

with $\varphi_{i}=\cos({\bm{W}}_{i}{\bm{x}}_{i}+{\bm{b}}_{i})$. Due to
$|\varphi_{i}|\leq 1$, we can derive (see Sec. 6 in supp. mat.)

$\displaystyle\|\nabla{\bm{u}}\|_{2}$ $\displaystyle\leq
d^{\frac{n}{2}}\left(\prod_{i=0}^{n}\|{\bm{W}}_{i}\|_{2}\right)\left(\prod_{i=0}^{n-1}\|{\varphi_{i}}({\bm{x}})\|_{2}\right)$
(7) $\displaystyle\leq d^{n}\cdot\prod_{i=0}^{n}\|{\bm{W}}_{i}\|_{2}$ (8)

in which $\|\cdot\|_{2}$ is the L2 norm, _i.e_. the largest singular value of
the matrix and Euclidean norm of the vector.

Consequently, the movement difference between two neighboring points in the
domain is constrained by Eq. (5) and Eq. (7). To increase the representation
power, one can straightforwardly increase the number of hidden layers, or the
dimension of hidden variables, because both can increase the upper-bound of
$\|\nabla{\bm{u}}\|_{2}$.

Our DOMA model can improve the model capacity without modifying the hidden
layers. Referring to Eq. (4), its Jacobian matrix is given by

$\frac{\partial{\bm{y}}}{\partial{\bm{x}}}={\bm{A}}+\langle\nabla{\bm{A}},{\bm{x}}\rangle+\nabla{\bm{u}}({\bm{x}}),$
(9)

which replaces the identity matrix in Eq. (5) with two complex terms. Since
the identity matrix in Eq. (5) does not contribute to the motion complexity,
our method intrinsically increases the complexity, while keeping the model
hidden layers unchanged. Consequently, more complex linear transformations
with additional DOFs, such as scaling and shearing, are introduced to every
infinitesimal area in the entire domain, thereby increasing the motion
complexity globally.

####  3.2.2 The Variants of DOMA

The motion complexity can be controlled by applying different constraints on
${\bm{A}}$, leading to different versions according to the DOFs, inspired by
existing works _e.g_. [45, 30, 27]. We denote the model Eq. (4) as DOMA-
Affinity. In our implementation, the SIREN network outputs a 12-dimensional
variable. The first 9 variables are reshaped to ${\bm{A}}$ and the rests are
regarded as ${\bm{u}}$.

DOMA-Trans. When ${\bm{A}}$ is an identity matrix, Eq. (4) degenerates to a
translation field, which formulated by

${\bm{y}}={\bm{x}}+{\bm{u}}({\bm{x}},t).$ (10)

This model can be regarded as a straightforward extension of DPF [49] to the
spatiotemporal domain.

DOMA-SE(3). With ${\bm{A}}={\bm{Q}}\in SO(3)$, _i.e_. a rotation matrix in the
3D space, Eq. (4) becomes to

${\bm{y}}={\bm{Q}}({\bm{x}},t){\bm{x}}+{\bm{u}}({\bm{x}},t).$ (11)

Besides producing the translation, we let the SIREN network output the 6D
continuous rotation representation [73], and perform orthogonalization to get
the rotation matrix.

DOMA-Scaled SE(3). By introducing an additional DOF for scaling, we can modify
Eq. (4) to

${\bm{y}}=s({\bm{x}},t){\bm{Q}}({\bm{x}},t){\bm{x}}+{\bm{u}}({\bm{x}},t),$
(12)

with $s({\bm{x}},t)$ being the spatiotemporal scalar field. In our
implementation, we let the SIREN network to produce an additional 1D variable,
and apply the softplus activation function [72, 13] to product
$s({\bm{x}},t)$.

####  3.2.3 Model Complexity Analysis

By changing DOFs at the SIREN output layer, the hidden layers remain
unchanged, and the model size is not linearly growing with the sequence
length. Provided the motion sequence has $T$ frames, DOMA-Affinity has
$16d+nd^{2}$ parameters, with $n$ and $d$ denoting the number of hidden layers
and the hidden dimensions, respectively. In contrast, the per-frame modeling
of DPF [49] requires $T-1$ SIRENs and has $(6d+nd^{2})(T-1)$ parameters in
total, leading to $\mathcal{O}(T)$ model size. Please see Tab. A1 in supp.
mat. for more details.

###  3.3 Motion Smoothness Regularization

Although introducing extra DOFs can improve the model representation power, it
increases the risk of overfitting. Without loss of generality, we assume the
motion field is piece-wise smooth in the domain. Referring to variational
methods in optical flow _e.g_. [74, 5], we introduce the following smoothness
regularization loss, _i.e_.

$\mathcal{L}_{H}=\mathbb{E}_{t,{\bm{x}}}\left[\Psi\left(\|\nabla{\bm{A}}\|_{F}^{2}+\|\nabla{\bm{u}}\|_{F}^{2}\right)\right],$
(13)

in which $\Psi(s^{2})=\sqrt{1+s^{2}}-1$ is the convex Charbonnier function [8]
to approximate the L1 norm, $\nabla(\cdot)$ denotes computing the spatial
gradients, and $\|\cdot\|_{F}$ denotes the Frobenius norm. Intuitively, the
local motion is parameterized by ${\bm{A}}$ and ${\bm{u}}$, and hence the zero
value of the above loss term suggests all points conduct the identical affine
transformation. The Charbonnier function plays the role of a robustifier, with
which the difference of motions between two neighboring points is less
penalized compared to L2 norm. With prior knowledge on the scene dynamics,
$\Psi(s^{2})$ can be changed to other terms, _e.g_. $\Psi(s^{2})=s^{2}$ to
encourage homogeneous motion.

Instead of employing auto-differentiation tools in _e.g_. PyTorch [46], we
implement analytical gradients of the SIREN network, _i.e_. Eq. (5), to speed
up computation. See Tab. A5 in supp. mat. for an empirical study.

##  4 Experiment

Without explicit mentioning, we set the first frame in the sequence as the
canonical frame, and normalize the time steps to $[-1,1]$. Please see supp.
mat. for more details and additional experiments.

###  4.1 Novel Point Motion Prediction

Based on a sparse set of observed point trajectories, we aim to predict the
motion of unseen points during training, in order to verify the quality of
learned dynamics.

Datasets. We select 7 sequences with various object categories, shapes, and
motions from the DeformingThings4D [28] dataset. For each sequence, we use 100
consecutive frames. We randomly select 25% mesh vertices for training the
motion field, and use the remaining ones for testing. In addition, we create
four synthetic sequences of elemental motions, _i.e_. translation, rotation,
scaling, and shearing, respectively, in order to investigate the model
representation power in detail. Each synthetic sequence has 20 frames and
contains 3000 points uniformly sampled from $[-1,1]^{3}$. Likewise, we
randomly select 25% of points for training, and leave the remaining ones for
testing.

Evaluation metrics. For evaluation, we employ the learned motion field to
transform testing points in the canonical frame to individual target frames.
We compute the scene flow end point error (EPE), _i.e_.
$\mathbb{E}_{t\in\\{1...,T\\}}[\|{\bm{v}}_{t}-{\bm{v}}^{gt}_{t}\|_{1}]$ and
${\bm{v}}_{t}={\bm{y}}_{t}-{\bm{x}}$, in which ${\bm{y}}_{t}$ denotes the
estimated corresponding point of ${\bm{x}}$ at time $t$. For
DeformingThings4D, we additionally use the learned motion field to warp the
canonical object mesh to each individual frame, sample $10^{6}$ points on both
the warped mesh and the ground truth mesh, and compute the Chamfer distance
[14] $\mathcal{L}_{CD}$ and the Chamfer normal distances $\mathcal{L}_{n}$, as
in [49, Table 2].

Baselines and ours. This task is highly related to learning the warping fields
in various scenarios, such as deformable object modeling [44], scene flow
estimation [26], and neural rendering [50, 45]. The warping field is commonly
parameterized with a neural network with ReLU activation functions [25] and
positional encodings [39]. Therefore, we leverage such kinds of neural
networks as baselines. Specifically, we denote MLP-ReLU as an MLP with ReLU
activation functions and 6 hidden layers of 128 dimensions. It takes the
concatenation of the 3D location and the 1D time step as input and outputs
motion vector. Additionally, we introduce positional encoding [39], or output
DCT coefficients [59], to create MLP-ReLU PE.6 and DCT-NeRF as baselines,
respectively. Moreover, we adapt the BANMO [67] deformation module into our
setting, and implement a modified version named BoneCloud, following the idea
of [42]. More details of these baseline methods are demonstrated in Sec. 8.1
of supp. mat.

We denote the DOMA models with their suffixes, _i.e_. -Trans, -SE(3), -Scaled
SE(3), and -Affinity, respectively. All their SIREN networks have 128 hidden
dimensions and 2 hidden layers. Moreover, we implement the elastic
regularization proposed in [45] to encourage rigid motion, which is denoted by
-E. Likewise, -H denotes our motion smoothness regularization without the
Charbonnier function, encouraging the motion is homogeneous.

Methods | EPE$\downarrow$ |  $\mathcal{L}_{CD}$ $\downarrow$ |  $\mathcal{L}_{n}$ $\downarrow$
---|---|---|---
MLP-ReLU [39] | 222.4 | 3.410 | 0.411
MLP-ReLU PE.6 [39] | 237.7 | 3.747 | 0.431
DCT-NeRF [59] | 215.0 | 3.766 | 0.347
BANMO [67] | 488.9 | 13.275 | 0.451
BoneCloud [67, 42] | 136.1 | 1.993 | 0.261
Ours-Trans | 78.5 | 1.401 | 0.215
Ours-SE(3) | 76.7 | 3.706 | 0.225
Ours-Scaled SE(3) | 76.2 | 2.074 | 0.220
Ours-Affinity | 78.1 | 1.266 | 0.218
Table 1: Results on DeformingThings4D sequences. EPE and $\mathcal{L}_{CD}$ are in $\times 10^{-4}$. Best results are in boldface.  Methods | Rotation | Scaling | Shearing | Translation
---|---|---|---|---
-Trans | 2725.4 | 1817.8 | 1619.5 | 1042.4
-SE(3) | 730.6 | 1991.4 | 1138.3 | 899.4
-Scaled SE(3) | 801.1 | 685.8 | 1524.7 | 1096.2
-Affinity | 1486.0 | 915.4 | 622.1 | 822.4
-Trans-E | 38.0 | 1669.6 | 753.6 | 38.8
-SE(3)-E | 20.0 | 1761.3 | 832.7 | 26.4
-scaled SE(3)-E | 21.2 | 1161.8 | 961.1 | 24.0
-Affinity-E | 19.2 | 155.7 | 864.0 | 15.7
-Trans-H | 4919.9 | 2056.4 | 2446.8 | 37.8
-SE(3)-H | 52.4 | 2012.4 | 1665.0 | 36.9
-scaled SE(3)-H | 29.3 | 22.1 | 688.0 | 30.3
-Affinity-H | 5.4 | 26.3 | 8.5 | 28.8
Table 2: Results on Synthetic sequences w.r.t. EPE (in $\times 10^{-4}$). ‘-E’
denotes the elasticity loss proposed in Nerfies [45], and ‘-H’ denotes our
smoothness loss.  Figure 2: Qualitative results on the Synthetic sequences.
The smoothness regularization is applied. Rows show types of motions, and
columns show the testing points in the canonical frame, a target frame, and
estimated results, respectively. See more descriptions in Sec. 4.1 and Sec.
8.1.3 in supp. mat.

Results. The results on the DeformingThings4D sequences are presented in Tab.
1. We can see that the SIREN-based methods achieve comparable performance, and
outperform other baseline methods consistently.

The results on the synthetic sequences are shown in Tab. A3 and Fig. 2. First,
we can see that the incorporated DOFs lead to different motion representation
behaviors. Methods with SE(3) transformations are more effective in
representing rotation and translation. By comparing ‘-Scaled SE(3)’ and
‘-SE(3)’, we can see the additional DOF benefits modeling scaling. The affine
transformation is effective in representing all cases. Second, we can see
appropriate regularization is important. The elasticity regularization
proposed in [45] is effective to encourage rigid motions, but performs
inferior if the motion is non-rigid, _i.e_. scaling and shearing. On the other
hand, our smoothness regularization significantly boosts the performance, if
the sequence fits the modelled DOFs.

Moreover, we find the structure of the linear matrix ${\bm{A}}$ and the
network hidden dimension have different behaviors to affect the model
representation power. Increasing the hidden dimension cannot simply introduce
new DOFs that the model can represent. see Sec. 8.1.3 in supp. mat. for
details.

###  4.2 Guided Mesh Alignment

Aligning a mesh template to a sequence of scans is important in various
applications, such as graphics [32] and healthcare [18]. We follow the guided
geometry deformation method of DPF [49]. Referring to Eq. 3, we minimize

$\sum_{t}\\{\alpha_{1}\mathcal{L}_{CD}\left({\bm{M}}_{t},{\bm{y}}({\bm{x}})\right)+\alpha_{2}\mathcal{L}_{V}({\bm{v}}^{c},{\bm{v}}^{t})+\alpha_{3}\mathcal{L}_{AIAP}\left({\bm{y}}({\bm{x}}),{\bm{x}}\right)+\alpha_{4}\mathcal{L}_{H}\\}.$
(14)

In this experiment, we set $(\alpha_{1},\alpha_{2})=(10^{3},1)$. The
regularization loss weights are $(\alpha_{3},\alpha_{4})=(1,0.001)$ if they
are enabled. Training terminates after 2000 iterations for all cases.

Datasets. We employ the Resynth dataset [32, 33], in which the human bodies
perform articulate motions, while the clothing, in particular the long skirt,
moves accordingly in a highly complex manner. We use 4 sequences from 4
individual subjects with different genders, body shapes, and clothing types.
Each sequence is downsampled by every 2 frames, and afterwards the first 30
frames are selected, leading to 16 sequences with 480 frames in total. We use
the SMPL-X [47] mesh vertices (10,475 points per frame) as the guidance
points, and the low-resolution scans (40,000 points per frame) as the targets
to fit. Furthermore, we perform Poisson surface reconstruction on the low-res
scan at the canonical frame, and obtain a mesh template with about 60K
vertices and 130K faces. During training, we learn the motion field based on
the guidance points and the low-res scans, and minimize Eq. (14). During
testing, we warp the mesh template vertices to individual frames based on the
learned motion field, and re-compute the vertex normals. Note these mesh
template are unseen during training.

Evaluation metrics. For evaluation, we compute the Chamfer distance of the
vertex locations and normals between the warped meshes with the target low-res
scans, _i.e_. $\mathcal{L}_{CD}$ and $\mathcal{L}_{n}$, as in Tab. 1 as well
as DPF [49, Table 2]. To verify the temporal smoothness, we additionally
compute two metrics: 1) The standard deviation (std) of edge lengths along the
temporal dimension, and report its maximal value. This metric is able to
reflect whether the mesh is significantly stretched or not. 2) The averaged
std of the velocity l2-norm of the mesh vertices, which measures the temporal
smoothness. These two metrics are denoted as STD(E) and STD(V), respectively.
Their values are the lower the better, but should not vanish because of the
conducted deformation and motion.

Baselines and ours. We compare the frame-wise DPF scheme that is suggested by
[49]. In addition, we investigate the effectiveness of the regularization loss
terms AIAP [49, Eq.13] and our proposed motion smoothness term. The method
notations are the same with Sec. 4.1. All SIREN networks have 3 hidden layers
and 128 hidden dimensions.

Figure 3: Illustration of results on two Resynth sequences. From left to right: The source mesh in the canonical frame, two consecutive frames of the target scans, the results from frame-wise DPF and DOMA-Affinity, respectively. Both AIAP [49] and smoothness regularization are applied. The bounding boxes highlight significant changes.  | $\mathcal{L}_{CD}\downarrow$ | $\mathcal{L}_{n}\downarrow$ | STD(E)$\downarrow$ | STD(V)$\downarrow$
---|---|---|---|---
DPF [49] | 1.149 | 0.122 | 11.6 | 24.6
-Trans | 1.230 | 0.128 | 12.8 | 22.9
-Affinity | 1.142 | 0.125 | 11.9 | 22.8
DPF-A [49] | 1.166 | 0.119 | 10.3 | 24.2
-Trans-A | 1.195 | 0.123 | 10.4 | 23.0
-Affinity-A | 1.151 | 0.122 | 10.6 | 23.0
DPF-H [49] | 1.142 | 0.123 | 10.3 | 24.2
-Trans-H | 1.207 | 0.128 | 10.8 | 22.9
-Affinity-H | 1.127 | 0.127 | 10.1 | 22.9
DPF-AH [49] | 1.189 | 0.120 | 9.3 | 24.3
-Trans-AH | 1.240 | 0.124 | 9.3 | 23.0
-Affinity-AH | 1.187 | 0.124 | 8.9 | 23.0
Table 3: Results of guided mesh alignment on our selected Resynth sequences. $\mathcal{L}_{CD}$ is in $\times 10^{-4}$. STD(E) and STD(V) are given in millimeters. ‘-A’ and ‘-H’ denote the AIAP regularization [49] and our smoothness regularization, respectively. ‘-AH’ denotes both regularization terms are applied. Best results are in boldface. Please see Tab. A7 for the performance of all models. | #params. | checkpoint size (KB)
---|---|---
DPF [49] | 1497600 | 7800
-Trans | 50048 | 139.6
-SE(3) | 50816 | 209.2
-Scaled SE(3) | 50944 | 209.8
-Affinity | 51200 | 210.8
Table 4: Evaluations on the model size on the Resynth sequence. Since
lightweight models are preferred, the numbers here are the lower the better.

Results. The results are shown in Tab. 3. Compared to frame-wise DPF, we can
see DOMA-Trans leads to consistently worse alignment accuracy, but better
temporal smoothness. This indicates the temporal regularity is obtained by
compressing the entire motion into a single SIREN-based network, whereas the
model’s representational power is not sufficient. When replacing the
translation field by an affinity field, _i.e_. DOMA-Affinity, the alignment
accuracy is consistently improved to a similar level with frame-wise DPF, and
the temporal smoothness is retained. This indicates that introduced extra DOFs
can effectively improve the model representation power. In addition, the AIAP
loss and the smoothness regularization can individually improve the
performances, but their combination does not lead to obvious advantages,
except for the edge length variations. Fig. 3 222Quantitatively, _i.e_.
($\mathcal{L}_{CD}$, $\mathcal{L}_{n}$, STD-E, STD-V) as in Tab. 3, DPF gives
$(3.962,0.229,16.0,35.7)$ and $(2.557,0.175,13.8,18.5)$ for the top and bottom
rows, respectively, whereas DOMA-Affinity gives $(3.208,0.216,19.8,29.3)$ and
$(2.532,0.169,10.5,15.6)$. illustrates some pairs of consecutive frames. We
can see that the frame-wise DPF scheme causes visible discontinuities and
artifacts in some regions, whereas the results of the affinity field have
better temporal regularity.

Furthermore, the advantage of DOMA can be reflected by the model compactness.
As shown in Tab. 4, DOMA models are significantly more lightweight. Adding
additional DOFs at the network output layer only increases the number of
parameters marginally.

##  5 Conclusion

In this work, we have advanced the DPF framework [49] into a continuous,
multi-frame affinity field model, which inherently exhibits spatiotemporal
regularity and improves representational capabilities without compromising
compactness. Incorporating the 1D time domain to the network input layer
ensures temporal regularity, and the DOFs at the output layer can manipulate
the model representation power without modifying the network hidden variables.
The experimental results on novel point motion prediction and guided mesh
alignment show its effectiveness and superiority to baselines.

Limitations and future works. First, we have 4 loss terms to minimize in the
task of guided mesh alignment, and inappropriate loss weights can degrade the
performance considerably. How to balance their weights is still not
transparent, which is worthy exploring in the future. Second, our method can
be employed to model warping fields for dynamic scene reconstruction and
rendering, which is not covered in this paper and will be studied as future
work. Note that our method requires corresponding points between frames.
Additional DOFs are effective to represent fine-grained movements, but might
behave as a disadvantage to extract corresponding points due to less
constraints. Third, our advanced model representation power is potential to
model highly complex dynamics, _e.g_. fluid fields, which can benefit specific
applications of medicine, aerodynamics, physics, _etc_. Furthermore, our model
is deterministic and does not consider the motion uncertainty. Thus, a future
direction is to develop a generative model on dynamics, which synthesizes
diverse dynamics based on the same set of point trajectories.

## Acknowledgement

We sincerely thank Shaofei Wang for fruitful suggestions, discussions, and
other helps. This project is partially supported by the SNSF grant 200021
204840.

## References

  * [1] Coding adventure: Simulating fluids.  https://github.com/SebLague/Fluid-Sim.
  * Behl et al. [2019] Aseem Behl, Despoina Paschalidou, Simon Donné, and Andreas Geiger.  Pointflownet: Learning representations for rigid motion estimation from point clouds.  In _CVPR_ , 2019.
  * Bogo et al. [2014] Federica Bogo, Javier Romero, Matthew Loper, and Michael J Black.  Faust: Dataset and evaluation for 3d mesh registration.  In _CVPR_ , 2014.
  * Bogo et al. [2017] Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  Dynamic FAUST: Registering human bodies in motion.  In _CVPR_ , 2017.
  * Brox et al. [2004] Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim Weickert.  High accuracy optical flow estimation based on a theory for warping.  In _ECCV_ , 2004.
  * Campbell et al. [2020] Dylan Campbell, Liu Liu, and Stephen Gould.  Solving the blind perspective-n-point problem end-to-end with robust differentiable geometric optimization.  In _ECCV_ , 2020.
  * Chafaı et al. [2009] Djalil Chafaı, Djalil Chafä, Olivier Guédon, Guillaume Lecue, and Alain Pajor.  Singular values of random matrices.  _Lecture Notes_ , 2009.
  * Charbonnier et al. [1994] Pierre Charbonnier, Laure Blanc-Feraud, Gilles Aubert, and Michel Barlaud.  Two deterministic half-quadratic regularization algorithms for computed imaging.  In _ICIP_ , 1994.
  * Chen et al. [2018] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.  Neural ordinary differential equations.  _NeurIPS_ , 2018.
  * Chen et al. [2021] Xu Chen, Yufeng Zheng, Michael J Black, Otmar Hilliges, and Andreas Geiger.  Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes.  In _ICCV_ , 2021.
  * Deng et al. [2022] Bailin Deng, Yuxin Yao, Roberto M Dyke, and Juyong Zhang.  A survey of non-rigid 3d registration.  In _Computer Graphics Forum_ , 2022.
  * Dosovitskiy et al. [2015] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox.  Flownet: Learning optical flow with convolutional networks.  In _ICCV_ , 2015.
  * Dugas et al. [2000] Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia.  Incorporating second-order functional knowledge for better option pricing.  _NeurIPS_ , 2000.
  * Fan et al. [2017] Haoqiang Fan, Hao Su, and Leonidas J Guibas.  A point set generation network for 3d object reconstruction from a single image.  In _CVPR_ , 2017.
  * Gharaee et al. [2023] Zahra Gharaee, Felix Järemo Lawin, and Per-Erik Forssén.  Self-supervised learning of object pose estimation using keypoint prediction.  In _ICLR_ , 2023.
  * Goel et al. [2020] Shubham Goel, Angjoo Kanazawa, and Jitendra Malik.  Shape and viewpoint without keypoints.  In _ECCV_ , 2020.
  * Groueix et al. [2018] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry.  3d-coded: 3D correspondences by deep deformation.  In _ECCV_ , 2018.
  * Hesse et al. [2019] Nikolas Hesse, Sergi Pujades, Michael J Black, Michael Arens, Ulrich G Hofmann, and A Sebastian Schroeder.  Learning and tracking the 3d body shape of freely moving infants from rgb-d sequences.  _PAMI_ , 2019.
  * Higham [2002] Nicholas J Higham.  _Accuracy and stability of numerical algorithms_.  2002\.
  * Jaderberg et al. [2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  _NeurIPS_ , 2015.
  * Kanazawa et al. [2018a] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In _CVPR_ , 2018a.
  * Kanazawa et al. [2018b] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik.  Learning category-specific mesh reconstruction from image collections.  In _ECCV_ , 2018b.
  * Karaev et al. [2023] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht.  Cotracker: It is better to track together.  _arXiv:2307.07635_ , 2023.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _NeurIPS_ , 2012.
  * Li et al. [2021a] Xueqian Li, Jhony Kaesemodel Pontes, and Simon Lucey.  Neural scene flow prior.  In _NeurIPS_ , 2021a.
  * Li and Harada [2022] Yang Li and Tatsuya Harada.  Non-rigid point cloud registration with neural deformation pyramid.  _NeurIPS_ , 2022.
  * Li et al. [2021b] Yang Li, Hikari Takehara, Takafumi Taketomi, Bo Zheng, and Matthias Nießner.  4dcomplete: Non-rigid motion estimation beyond the observable surface.  In _ICCV_ , 2021b.
  * Li et al. [2023] Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely.  Dynibar: Neural dynamic image-based rendering.  In _CVPR_ , 2023.
  * Lombardi et al. [2019] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh.  Neural volumes: Learning dynamic renderable volumes from images.  _ACM Trans. Graph._ , 2019.
  * Loper et al. [2014] Matthew Loper, Naureen Mahmood, and Michael J Black.  Mosh: Motion and shape capture from sparse markers.  _ACM Trans. Gr._ , 2014.
  * Ma et al. [2021a] Qianli Ma, Shunsuke Saito, Jinlong Yang, Siyu Tang, and Michael J Black.  Scale: Modeling clothed humans with a surface codec of articulated local elements.  In _CVPR_ , 2021a.
  * Ma et al. [2021b] Qianli Ma, Jinlong Yang, Siyu Tang, and Michael J Black.  The power of points for modeling humans in clothing.  In _ICCV_ , 2021b.
  * Mahmood et al. [2019] Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black.  AMASS: Archive of motion capture as surface shapes.  In _ICCV_ , 2019.
  * Mathis et al. [2020] Alexander Mathis, Steffen Schneider, Jessy Lauer, and Mackenzie Weygandt Mathis.  A primer on motion capture with deep learning: principles, pitfalls, and perspectives.  _Neuron_ , 2020.
  * Mihajlovic et al. [2021] Marko Mihajlovic, Yan Zhang, Michael J Black, and Siyu Tang.  Leap: Learning articulated occupancy of people.  In _CVPR_ , 2021.
  * Mihajlovic et al. [2022] Marko Mihajlovic, Shunsuke Saito, Aayush Bansal, Michael Zollhoefer, and Siyu Tang.  COAP: Compositional articulated occupancy of people.  In _CVPR_ , 2022.
  * Mihajlovic et al. [2024] Marko Mihajlovic, Sergey Prokudin, Marc Pollefeys, and Siyu Tang.  Resfields: Residual neural fields for spatiotemporal signals.  In _ICLR_ , 2024.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _ECCV_ , 2020.
  * Myronenko and Song [2010] Andriy Myronenko and Xubo Song.  Point set registration: Coherent point drift.  _PAMI_ , 2010.
  * Niemeyer et al. [2019] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Occupancy flow: 4d reconstruction by learning particle dynamics.  In _ICCV_ , 2019.
  * Novotny et al. [2022] David Novotny, Ignacio Rocco, Samarth Sinha, Alexandre Carlier, Gael Kerchenbaum, Roman Shapovalov, Nikita Smetanin, Natalia Neverova, Benjamin Graham, and Andrea Vedaldi.  Keytr: keypoint transporter for 3d reconstruction of deformable objects in videos.  In _CVPR_ , 2022.
  * Ovsjanikov et al. [2012] Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas.  Functional maps: a flexible representation of maps between shapes.  _ToG_ , 2012.
  * Palafox et al. [2021] Pablo Palafox, Aljaz Bozic, Justus Thies, Matthias Nießner, and Angela Dai.  Neural parametric models for 3d deformable shapes.  In _ICCV_ , 2021.
  * Park et al. [2021] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  _ICCV_ , 2021.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In _NeurIPS_ , 2019.
  * Pavlakos et al. [2019] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3D hands, face, and body from a single image.  In _CVPR_ , 2019.
  * Poppe [2007] Ronald Poppe.  Vision-based human motion analysis: An overview.  _Computer vision and image understanding_ , 2007.
  * Prokudin et al. [2023] Sergey Prokudin, Qianli Ma, Maxime Raafat, Julien Valentin, and Siyu Tang.  Dynamic point fields.  In _ICCV_ , 2023.
  * Pumarola et al. [2021] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In _CVPR_ , 2021.
  * Qian et al. [2024] Zhiyin Qian, Shaofei Wang, Marko Mihajlovic, Andreas Geiger, and Siyu Tang.  3dgs-avatar: Animatable avatars via deformable 3d gaussian splatting.  In _CVPR_ , 2024.
  * Rempe et al. [2021] Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J. Guibas.  Humor: 3d human motion model for robust pose estimation.  In _ICCV_ , 2021.
  * Saito et al. [2021] Shunsuke Saito, Jinlong Yang, Qianli Ma, and Michael J Black.  SCANimate: Weakly supervised learning of skinned clothed avatar networks.  In _CVPR_ , 2021.
  * Sitzmann et al. [2020a] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  _NeurIPS_ , 2020a.
  * Sitzmann et al. [2020b] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  In _NeurIPS_ , 2020b.
  * Spencer [2004] Anthony James Merrill Spencer.  _Continuum mechanics_.  2004\.
  * Tewari et al. [2022] Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al.  Advances in neural rendering.  In _Computer Graphics Forum_ , 2022.
  * Tsin and Kanade [2004] Yanghai Tsin and Takeo Kanade.  A correlation-based approach to robust point set registration.  In _ECCV_ , 2004.
  * Wang et al. [2021a] Chaoyang Wang, Ben Eckart, Simon Lucey, and Orazio Gallo.  Neural trajectory fields for dynamic novel view synthesis.  _arXiv preprint arXiv:2105.05994_ , 2021a.
  * Wang et al. [2023] Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely.  Tracking everything everywhere all at once.  In _ICCV_ , 2023.
  * Wang et al. [2021b] Shaofei Wang, Marko Mihajlovic, Qianli Ma, Andreas Geiger, and Siyu Tang.  Metaavatar: Learning animatable clothed human models from few depth images.  In _NeurIPS_ , 2021b.
  * Wang et al. [2022] Shaofei Wang, Katja Schwarz, Andreas Geiger, and Siyu Tang.  Arah: Animatable volume rendering of articulated human sdfs.  In _ECCV_ , 2022.
  * [63] Eric W. Weisstein.  ”matrix norm.” from mathworld–a wolfram web resource.  https://mathworld.wolfram.com/MatrixNorm.html.
  * Wen et al. [2023] Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Muller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield.  Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects.  In _CVPR_ , 2023.
  * Weng et al. [2022] Chung-Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher-Shlizerman.  Humannerf: Free-viewpoint rendering of moving people from monocular video.  In _CVPR_ , 2022.
  * Xu et al. [2020] Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, William T Freeman, Rahul Sukthankar, and Cristian Sminchisescu.  GHUM & GHUML: Generative 3D human shape and articulated pose models.  In _CVPR_ , 2020.
  * Yang et al. [2022] Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo.  Banmo: Building animatable 3d neural models from many casual videos.  In _CVPR_ , 2022.
  * Yuan et al. [2021] Ye Yuan, Shih-En Wei, Tomas Simon, Kris Kitani, and Jason Saragih.  Simpoe: Simulated character control for 3d human pose estimation.  In _CVPR_ , 2021.
  * Zhai et al. [2021] Mingliang Zhai, Xuezhi Xiang, Ning Lv, and Xiangdong Kong.  Optical flow and scene flow estimation: A survey.  _Pattern Recognition_ , 2021.
  * Zhang et al. [2021a] Siwei Zhang, Yan Zhang, Federica Bogo, Marc Pollefeys, and Siyu Tang.  Learning motion priors for 4d human body capture in 3d scenes.  In _ICCV_ , 2021a.
  * Zhang et al. [2021b] Yan Zhang, Michael J Black, and Siyu Tang.  We are more than our joints: Predicting how 3d bodies move.  In _CVPR_ , 2021b.
  * Zheng et al. [2015] Hao Zheng, Zhanlei Yang, Wenju Liu, Jizhong Liang, and Yanpeng Li.  Improving deep neural networks using softplus units.  In _IJCNN_ , 2015.
  * Zhou et al. [2019] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li.  On the continuity of rotation representations in neural networks.  In _CVPR_ , 2019.
  * Zimmer et al. [2011] Henning Zimmer, Andrés Bruhn, and Joachim Weickert.  Optic flow in harmony.  _IJCV_ , 2011.
  * Zuffi et al. [2017] Silvia Zuffi, Angjoo Kanazawa, David Jacobs, and Michael J. Black.  3D menagerie: Modeling the 3D shape and pose of animals.  In _CVPR_ , 2017.

Supplementary Material

###### Contents

  1. 1 Introduction
  2. 2 Related Work
  3. 3 Method
    1. 3.1 Preliminaries
      1. 3.1.1 SIREN [54]
      2. 3.1.2 Dynamic Point Field (DPF) [49]
    2. 3.2 DOMA: Spatiotemporal Affinity Motion Fields
      1. 3.2.1 On The Representation Power
      2. 3.2.2 The Variants of DOMA
      3. 3.2.3 Model Complexity Analysis
    3. 3.3 Motion Smoothness Regularization
  4. 4 Experiment
    1. 4.1 Novel Point Motion Prediction
    2. 4.2 Guided Mesh Alignment
  5. 5 Conclusion
  6. 6 Discussions on the Motion Model Bound
  7. 7 Additional Discussions on DOMA
    1. 7.1 The Network
    2. 7.2 Additional Discussions on Novelties
  8. 8 Experiment Details
    1. 8.1 Additional Presentations on Point Motion Prediction (Sec. 4.1)
      1. 8.1.1 Dataset
      2. 8.1.2 Baselines
      3. 8.1.3 More Results on the Synthetic Dataset
    2. 8.2 Additional Presentations on Guided Mesh Alignment (Sec. 4.2)
      1. 8.2.1 Dataset
      2. 8.2.2 Performances of All Model Variants
      3. 8.2.3 Analysis on Clothing Types
      4. 8.2.4 Influence of Hidden Dimensions
  9. 9 Additional Experiments
    1. 9.1 Learning 2D Image Deformation
    2. 9.2 Inferring Dynamics of Fluid Fields

##  6 Discussions on the Motion Model Bound

The singular values of the 3D linear transformation matrix carry essential
physical meanings. Via singular value decomposition, the motion can be
regarded as a consecutive operations of rotating to a new coordinate frame,
performing scaling in each dimension based on the singular values, and
rotating back to the original coordinate frame. Therefore, the upper bound of
such deformation is indicated by the largest singular value.

In the main paper, we demonstrate the representation power of DPF [49] is
bounded, based on Eq. (6) and Eq. (7). Here we present more details in terms
of a theorem with proof.

######  Theorem 1.

Provided

$\nabla{\bm{u}}={\bm{W}}_{n}\left(\prod_{i=0}^{n-1}{\bm{W}}_{i}\circ\varphi_{i}({\bm{x}})\right)$
(15)

and

$\varphi_{i}=\cos({\bm{W}}_{i}{\bm{x}}_{i}+{\bm{b}}_{i}),$ (16)

in which $\circ$ is the composition of element-wise multiplication and
broadcasting a vector to a matrix, as well as $i=\\{0,1,...,n-1\\}$, the bound
of the spectral norm of $\nabla{\bm{u}}$ is given by

$\|\nabla{\bm{u}}\|_{2}\leq d^{n}\cdot\prod_{i=0}^{n}\|{\bm{W}}_{i}\|_{2},$
(17)

in which $n$ and $d$ denote the number of hidden layers and the dimension of
hidden layers, respectively.

###### Proof.

Referring to the matrix norm properties [63, 19], we have the following
inequalities on the spectral norm, _i.e_.

$\displaystyle\|\nabla{\bm{u}}\|_{2}$
$\displaystyle=\left\|{\bm{W}}_{n}\left(\prod_{i=0}^{n-1}{\bm{W}}_{i}\circ\varphi_{i}({\bm{x}})\right)\right\|_{2}$
(18)
$\displaystyle\leq\|{\bm{W}}_{n}\|_{2}\cdot\prod_{i=0}^{n-1}\|{\bm{W}}_{i}\circ\varphi_{i}({\bm{x}})\|_{2}$
(19)
$\displaystyle\leq\|{\bm{W}}_{n}\|_{2}\left(\prod_{i=0}^{n-1}\|{\bm{W}}_{i}\|_{2}\right)\left(\prod_{i=0}^{n-1}\|\hat{\varphi}_{i}({\bm{x}})\|_{2}\right)$
(20)
$\displaystyle\leq\left(\prod_{i=0}^{n}\|{\bm{W}}_{i}\|_{2}\right)\left(\prod_{i=0}^{n-1}\|\hat{\varphi_{i}}({\bm{x}})\|_{2}\right),$
(21)

in which $\hat{\varphi}_{i}$ is the matrix with the same column $\varphi_{i}$,
and has the shape of $\mathbb{R}^{d\times q}$. This corresponds to the shape
of ${\bm{W}}_{i}$, and hence it has $q=3$ at the input layer and $q=d$ in the
hidden layers. Therefore, we assume $q=d$ in the following derivations to
obtain the upper bound.

Note the rank of the matrix $\hat{\varphi}_{i}$ is 1, and we can have

$\displaystyle\|\hat{\varphi_{i}}({\bm{x}})\|_{2}=\|\hat{\varphi_{i}}({\bm{x}})\|_{F}$
$\displaystyle=\sqrt{d}\cdot\|\varphi_{i}({\bm{x}})\|_{2}\leq d,$ (22)

according to $\|\varphi_{i}\|_{\infty}\leq 1$. Thus, we can derive

$\|\nabla{\bm{u}}\|_{2}\leq d^{n}\cdot\prod_{i=0}^{n}\|{\bm{W}}_{i}\|_{2}.$
(23)

∎

Although the constant factor $d^{n}$ is large, it is only reached when every
$|\varphi_{i}|$ is equal to 1, which is implausible in practice. In addition,
the entries in ${\bm{W}}_{i}$ are from the uniform distribution with a tiny
range around 0 [54], which further constrains the spectral norm of the
Jacobian matrix. Due to the challenges of spectral analysis on high-
dimensional random matrices, we can look into the degenerated 1D case, which
is given by

$\frac{du}{dx}=w_{n}\prod_{i=0}^{n-1}w_{i}\cos(w_{i}x+b_{i}).$ (24)

In this case, we can easily derive

$\displaystyle\left|\frac{du}{dx}\right|=\left|w_{n}\prod_{i=0}^{n-1}w_{i}\cos(w_{i}x+b_{i})\right|\leq\prod_{i=0}^{n}|w_{i}|,$
(25)

which indicates that the motion complexity is heavily bounded.

##### A statistical perspective.

The boundedness can be also investigated from a statistical perspective.
Starting with

$\|\nabla{\bm{u}}\|_{2}\leq\|{\bm{W}}_{n}\|_{2}\cdot\prod_{i=0}^{n-1}\|{\bm{W}}_{i}\circ\varphi_{i}({\bm{x}})\|_{2}$
(26)

that is from Eq. (18), we can reason the entries of
${\bm{W}}_{i}\circ\varphi_{i}({\bm{x}})$ are converging to the standard normal
distribution if the model weights are initialized as in SIREN [54].
Specifically, the entries of ${\bm{W}}_{i}$ are from the defined uniform
distribution, $\varphi_{i}$ is from the arcsine distribution, since the cosine
activation function is equivalent to the phase-shifted sine activation
function and the bias does not modify the distribution for high enough
frequency [54, Theorem 1.8]. According to [7, Theorem 2.5], the largest
singular value of ${\bm{A}}_{i}={\bm{W}}_{i}\circ\varphi_{i}({\bm{x}})$ is
bounded, having

$\lim_{d\to\infty}\sup\lambda_{1}(d^{-1}{\bm{A}}_{i}{\bm{A}}^{T}_{i})\leq 4,$
(27)

in which $d$ is the hidden dimension and $\lambda_{1}$ is the largest
eigenvalue. Therefore, their compositions with $i=0,\dots,n-1$ are also
bounded.

##  7 Additional Discussions on DOMA

###  7.1 The Network

##### The model architecture.

The DOMA models can be visualized in Fig. A1. In this case, the SIREN [54]
network contains one input layer, one output layer, and two hidden layers. In
the case of the ‘SE(3)’ and ‘scaled SE(3)’ models, 6D continuous rotation
representations [73] are produced by the output layer, which are then
orthogonalized to rotation matrices.

Figure A1: Illustration of the DOMA model architecture. The SIREN layers [54]
produce an affine transformation, which maps the point from ${\bm{x}}$ to
${\bm{y}}$ at time $t$.

##### The model sizes.

Since the 1D temporal dimension is incorporated in the input layer, DOMA
models have $\mathcal{O}(1)$ complexity w.r.t. the motion sequence length. The
sizes of different models are summarized in Tab. A1.

DPF | -Trans | -SE(3) | -Scaled SE(3) | -Affinity
---|---|---|---|---
$(6d+nd^{2})(T-1)$ | $7d+nd^{2}$ | $13d+nd^{2}$ | $14d+nd^{2}$ | $16d+nd^{2}$

Table A1: The number of parameters in the employed SIREN network. $T$, $n$,
$d$ denote the number of frames in the sequence, the number of network hidden
layers, and the hidden dimension, respectively. The suffixes denote different
versions of DOMA.

###  7.2 Additional Discussions on Novelties

Modeling the deformation field or the motion field is not a new task. Instead,
various methods have been developed within respective tasks, such as geometry
deformation, neural rendering, dynamic scene reconstruction, avatar creation,
etc. Their exploited motion methods are diverse in terms of the neural
architecture, positional encoding, underlying deformation models, and so on.
However, an important aspect is often overlooked: the motion field should be
spatiotemporally regularized by nature. To fill this gap, we leverage the
SIREN [55] network, and extend the start-of-the-art work DPF [49] to a multi-
frame smooth affinity field model. By introducing additional DOFs at the
output layer, we find the model representation power is improved in a
different way from enlarging the model hidden layers, and come up with a
solution to increase the model capacity while retaining the model size.
Moreover, we introduce a smoothness regularization term to overcome
overfitting, which does not assume the underlying motion is _e.g_. rigid like
in [45]. The effectiveness of DOMA is demonstrated with experiments in Sec. 4
and the supp. mat.

The advantage of DOMA is more obvious when the ground truth motion is more
complex. An example is modeling the loose long skirt motion. As shown in Tab.
A8, DOMA-Affinity is consistently superior to DPF on the ‘felice’ sequences of
Resynth. Another example is modeling the fluid dynamics, which is investigated
in Sec. 9.2. We can see DOMA-Affinity outperforms DPF significantly. Since DPF
only models deformations between the canonical frame and frame $t$, it cannot
ensure the temporal smoothness between $t$ and $t+1$.

Despite aiming at different tasks, our work is also related to object shape
and view recovery from images. Kanazawa _et al_. [22] propose a framework to
learn from an annotated image collection, and recover the 3D shape in a
canonical frame, the camera pose, and the texture of an object from a single
image. The 3D object shape is parameterized by a learned mean shape and per-
instance predicted deformation. To encourage additional properties such as
surface smoothness and regularized deformation, generic priors are leveraged
in the training loss. Goel _et al_. [16] extend this framework to learn from
an image collection without annotations of the keypoints and the camera. To
further improve the performance, Gharaee _et al_. [15] propose to predict a
set of keypoints to represent the shape, corresponding to positions on the
category-specific mean shape in 3D. Afterwards, the camera pose is estimated
by a robust PnP network [6]. These solutions of decoupling the instance-level
shape into the mean shape and the deformation also inspire us how to model
motions. Furthermore, we are encouraged by these works to reconstruct dynamic
scenes from multiview videos as future work.

##  8 Experiment Details

###  8.1 Additional Presentations on Point Motion Prediction (Sec. 4.1)

We leverage and modify the codebase of ResFields [38] for the implementations
of baselines MLP-ReLU and DCT-NeRF [59].

####  8.1.1 Dataset

The 7 sequences from DeformingThings4D [28] are listed in Tab. A2. For each
sequence, we extract the first 100 frames and regard the first frame as the
canonical frame.

bear3EP_Agression
---
demon_JazzDancing
dragonOLO_act25
michelle_StepHipHopDance
mutant_Defeated
tigerD8H_Swim17
vampire_Breakdance1990
vanguard_JoyfulJump
Table A2: The leveraged DeformingThings4D [28] sequences in Sec. 4.1.

####  8.1.2 Baselines

##### MLP-ReLU and MLP-ReLU PE.6.

MLPs with ReLU [25] are frequently used to warp points in existing works. In
our experiment, the architecture contains 6 hidden layers of 128 hidden
dimensions. In Tab. 1, the Fourier positional encoding [39] is not used in
‘MLP-ReLU’, but is applied in ‘MLP-ReLU PE.6’ with 6-level resolutions.

##### DCT-NeRF [59].

The Fourier positional encoding is not applied. Rather than outputting the
target point location ${\bm{y}}$, this baseline method produces the
coefficients of a DCT basis that is jointly learned from the data. Similar
technology is also employed in [29].

##### BANMO [67].

BANMO is a solution to reconstruct the avatar of a generic object, _e.g_. cat,
from a monocular video. The avatar bones are modelled by a set of 3D
Gaussians, and the skinning weight is a combination of a Gaussian-based
weighting function and a neural network. The 3D location of a query point is
encoded by Fourier encoding [39]. The rest pose code is derived by a linear
layer, and the pose code is derived by the Fourier encoding of the frame and a
linear layer. In our experiment, we adopt its avatar deformation module into
our setting and use the hyper-parameters as in the original paper [67].
Provided a set of training point trajectories, we optimize the 3D Gaussians
and the relevant networks as in [67]. During testing, we animate the testing
points in the canonical frame to produce the trajectories, based on the
learned Gaussians and networks.

##### BoneCloud.

Based on BANMO [67] and KeyTr [42], we propose this BoneCloud method, which is
a learnable bone basis. Compared to BANMO, this BoneCloud method does not
employ any nonlinear neural network. Instead, it has a point cloud in the
canonical frame, and each point stores a time sequence of SE(3)
transformations. The skinning weights are created by a pre-fixed radial basis
function. Consequently, a 3D point ${\bm{x}}$ in the canonical frame can be
transformed to ${\bm{y}}$ at frame $t$, via linear blend skinning.
Specifically, it is given by

$\begin{pmatrix}{\bm{y}}\\\
1\end{pmatrix}=\left(\sum_{k}w_{k}{\bm{T}}_{k}^{t}\right)\begin{pmatrix}{\bm{x}}\\\
1\end{pmatrix}$ (28)

and

$\displaystyle\tilde{w}_{k}$
$\displaystyle=\exp{(-\sigma\|{\bm{x}}-{\bm{v}}_{k}\|_{2})}$ (29)
$\displaystyle{w}_{k}$
$\displaystyle=\frac{\tilde{w}_{k}}{\sum_{k}\tilde{w}_{k}},$ (30)

in which ${\bm{v}}\in\mathbb{R}^{3}$ is a bone in the bone cloud, $k$ is the
index of the bone, ${\bm{T}}_{k}^{t}\in SE(3)$ denotes the transformation of
the bone $k$ at time $t$. In our experiment, we leverage 1024 points as bones.
During training, we leverage the provided point trajectories to optimize the
bone locations at the canonical frame and the bone transformations at
individual time steps. During testing, we transform the testing points in the
canonical frame to individual target frames, so as to produce the point
trajectories.

####  8.1.3 More Results on the Synthetic Dataset

Corresponding to Fig. 2 in the main paper, we show the qualitative results of
all DOMA variants in Fig. A2. We can see that the affinity field is able to
represent all explored linear transformations. This indicates the output layer
highly influences the motion types that the model can represent.

In order to investigate how the hidden dimension influences the DOF
representation, we increase the hidden dimension of DOMA-Trans from 128 to
256. The results are shown in Tab. A4. Without smoothness regularization, we
can see that a higher hidden dimension slightly improves the performance in
some cases, but degrades the performance on translation, probably due to
overfitting. When applying the smoothness regularization to overcome
overfitting, motion prediction on translation is significantly improved,
whereas the performances on other linear transformations are much worse. On
the other hand, the performances of DOMA-Affinity on all motion types are
consistently and considerably improved by the smoothness regularization. Based
on these observations, we can conclude that

  * •

Both the hidden dimension and the DOFs represented by ${\bm{A}}$ can influence
the model representation power.

  * •

Increasing the hidden dimension improves the performance but not always.
Overfitting could occur.

  * •

The smoothness regularization can improve the performance significantly if the
ground truth DOF is explicitly modeled at the output layer. Otherwise, it can
degrade the performance.

  * •

Increasing the hidden dimension cannot simply increase the DOF
representations. Otherwise, the smoothness regularization should lead to
consistent improvements for all linear transformations.

Methods | Rotation | Scaling | Shearing | Translation
---|---|---|---|---
-Trans | 2725.4 | 1817.8 | 1619.5 | 1042.4
-SE(3) | 730.6 | 1991.4 | 1138.3 | 899.4
-Scaled SE(3) | 801.1 | 685.8 | 1524.7 | 1096.2
-Affinity | 1486.0 | 915.4 | 622.1 | 822.4
-Trans-E | 38.0 | 1669.6 | 753.6 | 38.8
-SE(3)-E | 20.0 | 1761.3 | 832.7 | 26.4
-scaled SE(3)-E | 21.2 | 1161.8 | 961.1 | 24.0
-Affinity-E | 19.2 | 155.7 | 864.0 | 15.7
-Trans-H | 4919.9 | 2056.4 | 2446.8 | 37.8
-SE(3)-H | 52.4 | 2012.4 | 1665.0 | 36.9
-scaled SE(3)-H | 29.3 | 22.1 | 688.0 | 30.3
-Affinity-H | 5.4 | 26.3 | 8.5 | 28.8
Table A3: Results on Synthetic sequences w.r.t. EPE (in $\times 10^{-4}$).
‘-E’ denotes the elasticity loss proposed in Nerfies [45], and ‘-H’ denotes
our smoothness loss. Best results are in boldface.

Methods | Rotation | Scaling | Shearing | Translation
---|---|---|---|---
-Trans-128d | 2725.4 | 1817.8 | 1619.5 | 1042.4
-Trans-256d | 2187.1 | 1846.9 | 1515.8 | 1231.8
-Affinity-128d | 1486.0 | 915.4 | 622.1 | 822.4
-Trans-H-128d(0.1) | 4919.9 | 2056.4 | 2446.8 | 37.8
-Trans-H-256d(0.1) | 4911.8 | 2078.7 | 1733.7 | 217.4
-Trans-H-256d(1) | 10945.3 | 8400.4 | 8701.2 | 16.2
-Affinity-H-128d(0.1) | 5.4 | 26.3 | 8.5 | 28.8

Table A4: Results on Synthetic sequences as in Tab. A3 in the main paper.
Numbers denote EPE in $\times 10^{-4}$. ‘-128d’ and ‘-256d’ denote the hidden
dimension of the SIREN network. The number in $()$ denotes the weight of the
smoothness loss term. Best results are in boldface. Figure A2: Illustrations
of results on the Synthetic sequences. The smoothness regularization is
applied. Rows show types of motions, and columns show the testing points in
the canonical frame, a target frame, and estimated results from different
methods, respectively.

##### Runtime analysis.

In addition, we compare our derived analytical gradients with auto-diff of
Pytorch [46] w.r.t. the runtime. We set the smoothness loss weight to 0.1, and
train DOMA-Affinity for 1000 iterations. This experiment is conducted with
Ubuntu 20.04, NVIDIA TITAN RTX 24GB, CUDA 11.4, 32GB RAM. The results are
shown in Tab. A5. We can see the analytical gradients improve the efficiency
consistently. Compared to the standard auto-diff, the runtime is reduced by
28%.

Methods | Rotation | Scaling | Shearing | Translation | average
---|---|---|---|---|---
auto-diff [46] | 133.68 | 132.36 | 133.44 | 133.51 | 133.25
analytical grad | 96.08 | 96.26 | 95.78 | 95.92 | 96.01
Table A5: Comparison between our derived analytical gradients and auto-diff of
Pytorch. Runtime is measured in seconds.

###  8.2 Additional Presentations on Guided Mesh Alignment (Sec. 4.2)

####  8.2.1 Dataset

We employ the ReSynth dataset [32, 33] in this study. Specifically, we choose
16 sequences from 4 subjects in the packed sequences in the test split (see
Tab. A6). For each sequence, we first perform down-sampling by every 2 frames,
and then select the first 30 frames for experiments. The first frame in each
sequence is regarded as the canonical frame.

The motion complexity depends on the subject and the clothing type. As shown
in Fig. A3, sequences with ‘rp_felice_posed_004’ are more complex than others,
because of the loose long skirt. In this case, the points on the long skirt
are far away from the body surface, which is aligned and guided by the SMPL-X
[47] mesh vertices. Other subjects have tight clothes.

Figure A3: Illustrations of the 4 subjects in our employed sequences. These meshes are created by Poisson surface reconstruction based on the provided oriented points in the canonical frames. We have created 16 such meshes for individual sequences, which are roughly at the same pose, _i.e_. the A-pose. Subjects | Actions
---|---
rp_aaron_posed_002 |  | 96_jerseyshort_hips
---
96_jerseyshort_squats
96_longshort_flying_eagle
96_longshort_tilt_twist_left
rp_celina_posed_005 |  | 96_jerseyshort_hips
---
96_jerseyshort_squats
96_longshort_flying_eagle
96_longshort_tilt_twist_left
rp_felice_posed_004 |  | 96_jerseyshort_hips
---
96_jerseyshort_squats
96_longshort_flying_eagle
96_longshort_tilt_twist_left
rp_janna_posed_032 |  | 96_jerseyshort_hips
---
96_jerseyshort_squats
96_longshort_flying_eagle
96_longshort_tilt_twist_left
Table A6: Employed Resynth sequences in Sec. 4.2.

####  8.2.2 Performances of All Model Variants

In Tab. 3, we only show the performance of the DPF baseline, DOMA-Trans, and
DOMA-Affinity. Here we show the performances of all DOMA variants under the
same experiment setting. The results are presented in Tab. A7. We can draw
similar conclusions as in Sec. 4.2. The performance of the affinity field is
similarly better than other variants, in particular on the Chamfer distances.

| $\mathcal{L}_{CD}\downarrow$ | $\mathcal{L}_{n}\downarrow$ | STD(E)$\downarrow$ | STD(V)$\downarrow$
---|---|---|---|---
DPF [49] | 1.149 | 0.122 | 11.6 | 24.6
-Trans | 1.230 | 0.128 | 12.8 | 22.9
-SE(3) | 1.343 | 0.134 | 16.2 | 22.9
-Scaled SE(3) | 1.273 | 0.127 | 16.2 | 22.8
-Affinity | 1.142 | 0.125 | 11.9 | 22.8
DPF-A [49] | 1.166 | 0.119 | 10.3 | 24.2
-Trans-A | 1.195 | 0.123 | 10.4 | 23.0
-SE(3)-A | 1.278 | 0.123 | 11.5 | 23.0
-Scaled SE(3)-A | 1.20 | 0.120 | 11.3 | 23.0
-Affinity-A | 1.151 | 0.122 | 10.6 | 23.0
DPF-H [49] | 1.142 | 0.123 | 10.3 | 24.2
-Trans-H | 1.207 | 0.128 | 10.8 | 22.9
-SE(3)-H | 1.230 | 0.127 | 12.2 | 23.0
-Scaled SE(3)-H | 1.189 | 0.125 | 11.5 | 22.9
-Affinity-H | 1.127 | 0.127 | 10.1 | 22.9
DPF-AH [49] | 1.189 | 0.120 | 9.3 | 24.3
-Trans-AH | 1.240 | 0.124 | 9.3 | 23.0
-SE(3)-AH | 1.265 | 0.124 | 10.9 | 23.1
-Scaled SE(3)-AH | 1.255 | 0.126 | 8.7 | 23.0
-Affinity-AH | 1.187 | 0.124 | 8.9 | 23.0
Table A7: Results of guided mesh alignment on our selected Resynth sequences.
$\mathcal{L}_{CD}$ is in $\times 10^{-4}$. STD(E) and STD(V) are given in
millimeters. This table is supplementary to Tab. 3 in the main paper.

####  8.2.3 Analysis on Clothing Types

In addition to the averaged performance on all sequences, we have also
observed consistent trends on individual subjects that have different clothing
types. In this experiment, we leave ‘rp_felice_posed_004’ out of others, and
perform evaluations separately. For compactness, we only show the comparison
between our proposed affinity field and the frame-wise DPF models [49], with
the weights of the AIAP loss term and the motion smoothness term being
$(1,0.001)$. The results are shown in Tab. A8. We can see that the affinity
field outperforms DPF [49] on the subject with the long skirt, whereas
performs worse on subjects with tight clothing. A probable reason is that
points that are close to the body surface can be effectively guided by the
SMPL-X mesh vertices. Due to much more model parameters, the frame-wise DPF
models can overfit to the guidance points, and hence produces better results
on the body surfaces and the tight clothes. Simultaneously, it produces more
artifacts and discontinuities at regions that are far away from the guidance
points, leading to inferior performance to the affinity field.

Subjects | Methods | CD$\downarrow$ | CDN$\downarrow$ | STD(E)$\downarrow$ | STD(V)$\downarrow$
---|---|---|---|---|---
rp_felice_posed_004 |  DPF-AH [49] | 3.086 | 0.194 | 14.4 | 25.8
-Affinity-AH | 2.857 | 0.190 | 15.3 | 21.9
others |  DPF-AH [49] | 0.557 | 0.096 | 7.6 | 23.8
-Affinity-AH | 0.630 | 0.102 | 6.8 | 23.3
Table A8: Evaluation of methods on the Resynth sequence ‘rp_felice_posed_004’
as discussed in Section 4.2.

Subjects | Methods | CD$\downarrow$ | CDN$\downarrow$ | STD(E)$\downarrow$ | STD(V)$\downarrow$
---|---|---|---|---|---
all sequences |  DPF-AH [49] | 1.047 | 0.100 | 10.1 | 24.2
-Trans-AH | 1.058 | 0.106 | 10.6 | 23.2
-SE(3)-AH | 1.055 | 0.105 | 10.9 | 23.2
-Scaled SE(3)-AH | 1.060 | 0.104 | 10.3 | 23.2
-Affinity-AH | 1.023 | 0.105 | 9.8 | 23.2
rp_felice_posed_004 |  DPF-AH [49] | 2.740 | 0.151 | 15.6 | 25.3
-Trans-AH | 2.724 | 0.157 | 16.0 | 22.6
-SE(3)-AH | 2.683 | 0.149 | 18.5 | 22.5
-Scaled SE(3)-AH | 2.694 | 0.148 | 17.5 | 22.6
-Affinity-AH | 2.544 | 0.152 | 16.5 | 22.5
others |  DPF-AH [49] | 0.483 | 0.083 | 8.3 | 23.8
-Trans-AH | 0.503 | 0.089 | 8.8 | 23.4
-SE(3)-AH | 0.513 | 0.090 | 8.4 | 23.5
-Scaled SE(3)-AH | 0.516 | 0.089 | 7.9 | 23.5
-Affinity-AH | 0.515 | 0.090 | 7.6 | 23.5

Table A9: Evaluations based on the models with 256D hidden variables. Other
settings are identical with Tab. 3 and A8. Best results are highlighted in
boldface.

####  8.2.4 Influence of Hidden Dimensions

We set the hidden dimension to 128 by default in the main paper and the above
experiments. Here we increase it to 256 and re-evaluate the performances.
According to our analysis of the motion model bound (see Sec. 3 and 6),
increasing the hidden dimension is able to improve the model representation
power on the motion complexity.

The results are presented in Tab. A9. Compared to models with 128D hidden
variables (see Tab. 3 and Tab. A8), models with 256D hidden variables
consistently produce better results. With this new setting, the performance
gaps between individual methods tend to vanish. The temporal smoothness tends
to degrade though.

In the meanwhile, we can see that the affinity field still has comparably
better performance than frame-wise DPF [49], but produces smoother results,
leading to the same observation and conclusion as demonstrated in Sec. 4.2.
Focusing on the performances on different sequences, we can see DPF [49] still
outperforms DOMA models on tight clothing w.r.t. alignment, but the gap
becomes smaller compared to Tab. 3. The affinity field outperforms DPF on the
loose long skirt sequence by a large margin. Furthermore, from the model size
perspective, our DOMA models are still significantly more lightweight than the
DPF [49] baseline.

##  9 Additional Experiments

Figure A4: Illustration of modeling 2D image deformation, in which the hidden dimension is 64. ‘GT’, ‘trans’, and ‘affine’ denote the ground truth, DOMA-Trans, and DOMA-Affinity, respectively. | Rotation | Scaling | Shearing | Translation
---|---|---|---|---
| -Trans | -Affinity | -Trans | -Affinity | -Trans | -Affinity | -Trans | -Affinity
hdim=32 | 60.8 | 85.0 | 31.2 | 8.9 | 11.7 | 5.4 | 100.4 | 44.0
hdim=64 | 64.5 | 33.9 | 15.3 | 6.2 | 10.5 | 4.3 | 25.8 | 18.7
hdim=128 | 35.4 | 20.3 | 8.2 | 4.6 | 6.4 | 3.8 | 20.2 | 20.4
Table A10: Evaluations in the 2D image deformations. As in Tab. A3, the
numbers denote EPE in $\times 10^{-4}$, and are the lower the better.

###  9.1 Learning 2D Image Deformation

Similar to the experiments on the 3D synthetic dataset, here we conduct an
experiment on 2D image deformation, in order to further investigate the
representation power of DOMA models.

##### Data, evaluation, and methods.

We use a RGB image of cat that has 512$\times$512 of pixels. As our synthetic
dataset, we perform translation, rotation, scaling, and shearing in the 2D
domain, and produce 30 frames. In each case, we randomly choose 25% points for
training the motion field, and use the remaining for testing. The evaluation
metric is the same as in Sec. 4.1. DOMA-Trans and DOMA-Affinity with different
hidden dimensions are applied in this experiment. No regularization is used
during training.

##### Results.

The quantitative evaluation is shown in Tab. A10, and some qualitative results
are shown in Fig. A4. We can see that the affinity model outperforms the
translation model consistently with different hidden dimensions. In
particular, the performance of the affinity model is superior when the hidden
dimension is smaller. These results demonstrate the advantages of additional
DOFs.

Figure A5: The particle system to simulate a fluid field in Unity3D, which is
implemented based on [1].

###  9.2 Inferring Dynamics of Fluid Fields

In Sec. 4.1, we have investigated the model representation power based on the
DeformingThings4D [28] sequences. Despite various model shapes and movements,
they are limited to elastic deformations of solid objects. In this section, we
propose a more challenging scenario, modeling a fluid field. To perform
empirical studies, we follow [1] to simulate how liquid moves in a bounded
field with Unity3D, and record the particle trajectories (see Fig. A5).

The entire sequence contains 931 frames and 27,000 particles. We randomly
choose 50% for training the motion field and use the rests for testing. We
find that all methods investigated in this paper are not able to reconstruct
the entire sequence. Thus, we down-sample the entire sequence by every 2
frames, and then trim the down-sampled sequence into 10-frame clips.
Specifically, the frame indices of the clips are
$\\{(t,t+10)\\}_{t=10,15,\ldots,325}$, in which the sequences with trivial
motions, _e.g_. static state in the beginning and steady state in the end, are
excluded. This pre-processing will lead to 22 clips in total.

In each clip, the first frame is regarded as the canonical frame. Points in
the canonical frame are transformed into individual target frames, and their
averaged L1 distances to the ground truth are minimized during training. The
evaluation metric is the scene flow end point error (EPE), identical to Sec.
4.1. We use the Adam optimizer [24] for training. The initial learning rate is
$1e-4$. Training terminates after 5000 epochs.

In this experiment, we compare frame-wise DPF [49], DOMA-Trans, and DOMA-
Affinity. We set these models to have 2 hidden layers, and set their hidden
dimensions to be 64D or 128D. Results are presented in Tab. A11. We can see
that these two spatiotemporal motion field models considerably outperform the
DPF baseline. In addition, the affinity model performs comparably better than
the translation field model. With fewer hidden variables, the advantage of the
additional DOFs are more obvious. Fig. A6 illustrates some examples of how
these methods perform. We can see that the frame-wise DPF method can lead to
significant discontinuities between frames, and less accurate motion
prediction than DOMA.

| 64D | 128D
---|---|---
DPF [49] | 400.20 | 501.60
DOMA-Trans | 176.43 | 188.22
DOMA-Affinity | 168.89 | 183.33
Table A11: Motion prediction of unseen points on fluid simulation sequences.
The numbers are EPEs in $\times 10^{-4}$. Results of 64D and 128D hidden
dimensions are both presented. Figure A6: Illustrations of three fluid
dynamics estimation results. The hidden variables have 64 dimensions. The
first three rows denote three sequences. Columns from left to right denote the
first frame, the last frame of ground truth, DOMA-Affinity, DOMA-Trans, and
frame-wise DPF, respectively. The zoomed-in regions in the third row are shown
below their respective images.