# 4D Atlas: Statistical Analysis of the Spatiotemporal Variability in
Longitudinal 3D Shape Data

Hamid Laga, Marcel Padilla, Ian H. Jermyn, Sebastian Kurtek, Mohammed
Bennamoun  Anuj Srivastava  Hamid Laga is with the Information Technology
Discipline, Murdoch University, Murdoch, 6150 (Australia), with the Harry
Butler Institute, Murdoch University, Murdoch, 6150 (Australia), Email:
<EMAIL_ADDRESS>and with the University of South Australia, The
Phenomics and Bioinformatics Research Centre, SA 5000, Australia. Marcel
Padilla is with TU Berlin, Germany. Email<EMAIL_ADDRESS>Ian H.
Jermyn is with Durham University. Email<EMAIL_ADDRESS>Sebastian
Kurtek is with the Ohio State University, US. Email<EMAIL_ADDRESS>Mohammed Bennamoun is with the University of Western Australia, Perth, WA
6009, Australia. Email<EMAIL_ADDRESS>Anuj Srivasta is with
Florida State University, US. Email<EMAIL_ADDRESS>Manuscript received
XXXX, 2021; revised XXX, 2021.

###### Abstract

We propose a novel framework to learn the spatiotemporal variability in
longitudinal 3D shape data sets, which contain observations of objects that
evolve and deform over time. This problem is challenging since surfaces come
with arbitrary parameterizations and thus, they need to be spatially
registered. Also, different deforming objects, also called _4D surfaces_ ,
evolve at different speeds and thus they need to be temporally aligned. We
solve this spatiotemporal registration problem using a Riemannian approach. We
treat a 3D surface as a point in a shape space equipped with an elastic
Riemannian metric that measures the amount of bending and stretching that the
surfaces undergo. A 4D surface can then be seen as a trajectory in this space.
With this formulation, the statistical analysis of 4D surfaces can be cast as
the problem of analyzing trajectories embedded in a nonlinear Riemannian
manifold. However, performing the spatiotemporal registration, and
subsequently computing statistics, on such nonlinear spaces is not
straightforward as they rely on complex nonlinear optimizations. Our core
contribution is the mapping of the surfaces to the space of Square-Root Normal
Fields (SRNF) where the $\mathbb{L}^{2}$ metric is equivalent to the partial
elastic metric in the space of surfaces. Thus, by solving the spatial
registration in the SRNF space, the problem of analyzing 4D surfaces becomes
the problem of analyzing trajectories embedded in the SRNF space, which has a
Euclidean structure. In this paper, we develop the building blocks that enable
such analysis. These include: (1) the spatiotemporal registration of
arbitrarily parameterized 4D surfaces even in the presence of large elastic
deformations and large variations in their execution rates; (2) the
computation of geodesics between 4D surfaces; (3) the computation of
statistical summaries, such as means and modes of variation, of collections of
4D surfaces; and (4) the synthesis of random 4D surfaces. We demonstrate the
performance of the proposed framework using 4D facial surfaces and 4D human
body shapes.

###### Index Terms:

Dynamic surfaces, Elastic metric, Square-Root Normal Field, Statistical
summaries, Shape synthesis and generation, 4D surface, Human4D, Face4D.

##  1 Introduction

Shape, an essential property of natural and man-made 3D objects, deforms over
time as a result of many internal and external factors. For instance,
anatomical organs such as bones, kidneys, and subcortical structures in the
brain deform due to natural growth or disease progression; human faces deform
as a consequence of talking, executing facial expressions, and aging.
Similarly, actions and motions such as walking, jumping, and running are the
result of deformations, over time, of the human body shape. The ability to
understand and model (1) the typical deformation patterns of a class of 3D
objects and (2) the variability of these deformations within and across object
classes has many applications. In medical diagnosis and biological growth
modeling, one is interested in measuring the intensity of pain from facial
deformations [1], and in distinguishing between normal growth and disease
progression using the deformation over time of body shape. In computer vision
and graphics, the ability to statistically model such spatiotemporal
variability can be used to summarize collections of 3D animations, and
synthesize and simulate animations and motions. Similar to 3D morphable face
models [2], these tools can also be used in a generative model for
synthesizing large corpora of labeled longitudinal 3D shape data, e.g., 4D
faces, for training deep neural networks.

This paper proposes a novel framework for the statistical analysis of
longitudinal 3D shape data composed of objects that deform over time. Each
object is represented as a closed manifold surface. We refer to an object
captured at different points in time, e.g., a 3D human face performing a
facial expression or speaking a sentence, or a 3D human body shape growing or
performing actions, as a _4D (or 3D + t) surface_. Given a set of 4D surfaces,
our goal is to:

  * •

Compute the mean deformation pattern, _i.e.,_ the statistical mean 4D surface.
For example, the same person can smile in different ways. Similarly, different
people smile differently. The goal is to learn, based on observed longitudinal
shape data, the typical smile.

  * •

Compute the directions of variation, analogous to Principal Component Analysis
(PCA) for modeling 3D shape variability [3, 4], but here we focus on modeling
variability in 4D surface collections.

  * •

Characterize a population of 4D surfaces using statistical models.

  * •

Synthesize new 4D surfaces by sampling, randomly or in a controlled fashion,
from these statistical models.

We refer to these tasks as the process of constructing a 4D atlas. Achieving
this goal requires solving important fundamental challenges. In fact, 3D
objects such as faces, human body shapes, and anatomical organs, which come
with arbitrary parameterizations, exhibit large elastic deformations within
the same object and across different objects. This makes their spatial
registration, _i.e.,_ finding one-to-one correspondences between each pair of
shapes, very challenging. In the case of 4D surfaces, there is an additional
temporal variability due to different execution rates (speeds) of evolution
within and across objects. For instance, a walking action can be executed at
variable speeds even by the same person. Thus, the statistical analysis of the
spatiotemporal variability in samples of 4D surfaces requires efficient
spatiotemporal registration of these samples. _Spatial registration_ refers to
the process of finding a one-to-one correspondence between two 3D surfaces of
the same individual, captured at different points in time, or of different
individuals. _Temporal registration_ refers to the problem of finding the
optimal time warping that aligns 4D surfaces, e.g., walking actions, performed
at different execution rates.

We treat a 4D surface as a trajectory in a high-dimensional nonlinear space.
We then formulate the problem of analyzing the spatiotemporal variability of
4D surfaces as the statistical analysis of elastic trajectories, where
elasticity corresponds to variations in the execution rates of the 4D
surfaces. However, performing statistics on trajectories embedded in nonlinear
spaces of high dimension is computationally expensive since it relies on
nonlinear optimizations. Our core contribution in this paper is the mapping of
the surfaces to the space of Square-Root Normal Fields (SRNF) [5, 4], which
has a Euclidean structure (see Section 3.1—in particular, the $\mathbb{L}^{2}$
metric in the space of SRNFs is equivalent to the partial elastic metric in
the space of surfaces), meaning that the problem of analyzing 4D surfaces
becomes the problem of analyzing trajectories, or curves, embedded in the
Euclidean space of SRNFs.

This paper develops the building blocks that enable such analysis. We use
these building blocks to compute statistical summaries, such as means and
modes of variation of collections of 4D surfaces, and for the synthesis,
either randomly or in a controlled fashion, of 4D surfaces. We demonstrate the
utility and performance of the proposed framework using 4D facial surfaces
from the VOCA dataset [6], 4D human body shapes from the Dynamic FAUST
(DFAUST) dataset [7], and dressed 4D human body shapes from the CAPE dataset
[8]. Our approach is, however, general and applies to all spherically-
parameterized surfaces. In summary, the main contributions of this article
are:

  * •

We represent 4D surfaces as trajectories in the space of SRNFs, which has a
Euclidean structure (Section 3.1). This key contribution enables the usage of
standard computational tools for the analysis and modeling of 4D surfaces
(Section 3.2).

  * •

We propose efficient algorithms for the spatiotemporal registration of 4D
surfaces and the computation of geodesics between such 4D surfaces, even in
the presence of large elastic deformations and significant variations in
execution rates (Sections 3.2.2 and 3.2.3).

  * •

The framework does not explicitly or implicitly assume that the
correspondences between the surfaces are given. It simultaneously solves for
the spatial and temporal registrations, and for the 4D geodesics that are
optimal under the proposed metrics.

  * •

We develop computational tools for (1) computing summary statistics of 4D
surfaces and (2) synthesizing 4D surfaces from formal statistical models
(Section 4).

The remainder of this paper is organized as follows. We first discuss related
work in Section 2. Section 3 describes the proposed mathematical framework.
Section 4 discusses its application to various statistical analysis tasks.
Section 5 presents the results and discusses the performance of the proposed
framework. Section 6 summarizes the main findings of this paper and discusses
future research directions.

##  2 Related work

We classify the state-of-the-art into two categories. Methods in the first
category focus on cross-sectional shape data (Section 2.1). Methods in the
second category focus on longitudinal shape data (Section 2.2).

###  2.1 Statistical models of cross-sectional 3D shape data

Modeling shape variability in 2D and 3D objects has been studied extensively
in the literature. Early methods use Principal Component Analysis (PCA) to
characterize the shape space of objects. Initially introduced for the analysis
of planar shapes, the active shape model of Cootes _et al._ [9] has been
extended to 3D faces [10] and 3D human bodies [3]; see [2] for a detailed
survey. These methods represent 3D objects as discrete sets of landmarks,
e.g., vertices, which are assumed to be in correspondence across a population
of objects, and use standard Euclidean metrics for their comparison. Thus,
they are limited to 3D objects that undergo small elastic deformations.

To handle large nonlinear variations, e.g., elastic deformations such as the
bending and stretching observed in 3D human body shapes, Anguelov _et al._
[11] introduced SCAPE, which represents body shape and pose-dependent shape in
terms of triangle deformations instead of vertex displacements. Hasler _et
al._ [12] learn two linear models: one for pose and one for body shape. Loper
_et al._ [13] introduce SMPL, a vertex-based linear model for human body shape
and pose-dependent shape variation. This model has been extensively used in
the literature for the analysis of the human body shape. It has also been
adapted to other types of objects such as animals [14] and human body parts
[15]. While these models can capture large variations, they exhibit two
fundamental limitations. First, they rely on separate models for pose-
independent shape, pose-dependent shape, and pose. Thus, they are limited to
specific classes of objects, e.g., human bodies. Changing the target
application, e.g., to animals [14] or infants [16], requires redefining the
model. Second, they either assume a given registration between the surfaces of
the 3D objects or solve for registration separately by matching vertices
across the surfaces using an unrelated optimization criterion.

Recently, there has been a growing interest in analyzing variability in 3D
shape collections using tools from differential and Riemannian geometry [17,
18, 5, 19, 20, 21, 4]; see [22] for a detailed survey. The work most relevant
to ours is the Square-Root Normal Field (SRNF) representation introduced in
[5]. In this work, parameterized surfaces are compared using a partial elastic
Riemannian metric defined as a weighted sum of a bending term and a stretching
term. More importantly, Jermyn _et al._ [5] show that by carefully choosing
the weights of these two terms, the complex partial elastic metric reduces to
the $\mathbb{L}^{2}$ metric in the space of SRNFs. Thus, by treating shapes of
objects as points in the SRNF space, a straight line between two points in
this space is equivalent to the geodesic (or shortest) curve in the original
space of surfaces under the partial elastic metric, and represents the optimal
deformation between them. As a result, one can perform statistical analysis in
the SRNF space using standard vector calculus, and then map the results back
to the space of surfaces (for visualization), using the approach of Laga _et
al._ [4]. Another important property of SRNFs is that both registration and
optimal deformation (geodesic) are computed jointly, using the same partial
elastic metric.

One of the fundamental problems in statistical shape analysis is
correspondence and registration, see [23]. Past methods do not define a shape
space and a metric that enable the computation of geodesics and statistics.
Also, correspondence methods that are based on the intrinsic properties of
surfaces, e.g., Generalized Multidimensional Scaling [24], spectral
descriptors [25], or functional maps (which rely on the availability of
descriptors) [26, 27], are primarily suited for surfaces that deform in an
isometric manner. They also require landmarks to resolve symmetry ambiguities.

###  2.2 Statistical models for longitudinal shape data

As stated in [7], we live in a 4D world of 3D shapes in motion. With the
availability of a variety of range sensing devices that can scan dynamic
objects at high temporal frequency, there is a growing interest in capturing
and modeling the 4D dynamics of objects [28, 29, 30]. For instance, Wand _et
al._ [28] and Tevs _et al._ [30] propose methods to reconstruct the deforming
geometry of time-varying point clouds. Li _et al._ [31] use sequences of 4D
scans to learn a statistical 3D facial model. This model, referred to as
FLAME, has been later used by Cudeiro _et al._ [6] to capture, learn, and
synthesize 3D speaking styles. Bogo _et al._ [7] build a 4D human dataset by
registering a 3D human template to sequences of 3D human scans performing
various types of actions. These methods focus on the 3D reconstruction of
deforming objects. The literature on the statistical analysis of their
spatiotemporal variability is rather limited.

Early works focused on longitudinal 2D shape data. For instance, Anirudh _et
al._ [32] represent the contour of planar shapes that evolve as trajectories
on a Grassmann manifold. They then use the Transported Square-Root Vector
Fields (TSRVF) representation for their rate-invariant analysis. This approach
was later extended to the analysis of the trajectories of sparse features or
landmarks measured on the surface of a deforming 3D object. For instance,
Akhter _et al._ [33] introduced a bilinear spatiotemporal basis to model the
spatiotemporal variability in 4D surfaces. The approach treats surfaces as $N$
discrete landmarks and uses the $\mathbb{L}^{2}$ metric and PCA in 4N for
their analysis. Thus, the approch is not suitable for highly articulated
shapes that undergo large articulated and elastic motion (e.g., human bodies).
The approach also assumes that the landmarks are in correspondence, both
spatially and temporally.

Anirudh _et al._ [32] and Ben Amor _et al._ [34] represent human body actions
using dynamic skeletons. By treating each skeleton, represented by a set of
landmarks, as a high-dimensional point on Kendall’s shape space [35], motions
become trajectories in a high-dimensional, Euclidean space. Thus, one can use
the rich literature on the statistical analysis of high-dimensional curves
[36] to build a framework for the statistical analysis of human motions and
actions. This approach, however, has two fundamental limitations. First, the
$\mathbb{L}^{2}$ metric on Kendall’s shape space is not suitable for large
articulated motions. Second, skeletons and landmarks do not capture surface
elasticity, and thus, cannot be used to model growth processes and surface
deformations due to motion. While this can be addressed by using two separate
models, one for shape and another for motion, it will fail to capture motion-
dependent shape variations.

Using the LDDMM framework [37], Debavelaere _et al._ [38] and Bone _et al._
[39] represent a 4D surface as a flow of deformations of the 3D volume around
each surface and then code deformations as geodesics on a Riemannian manifold.
However, in general, natural deformations do not correspond to geodesics but
can be arbitrary paths on the shape space. Also, deforming 3D volumes is
expensive in terms of computation and memory requirements. Finally, this
approach relies on manually specified landmarks to efficiently register the 3D
volumes. Our approach, which can handle large articulated and elastic motions,
works directly on surfaces, does not assume that deformations are (piecewise)
geodesics, and does not rely on landmarks for the spatiotemporal registration.

##  3 Mathematical framework

We describe in this section the proposed mathematical framework for the
spatiotemporal registration and comparison of 4D surfaces. Section 4 discusses
its application to various statistical analysis tasks. A 4D surface, where the
fourth dimension refers to time, is a 3D surface that evolves over time.
Examples of such 4D surfaces include facial expressions (e.g., a smiling
face), a human body shape performing an action such as walking or jumping, or
an anatomical organ that evolves over time due to natural growth or disease
progression. It can be represented as a path $\alpha(t),\ t\in[0,1]$ such that
$\alpha(0)$ and $\alpha(1)$ are the initial and final surfaces, respectively,
and $\alpha(t),\ 0<t<1$ are the intermediate surfaces. The main challenges
posed by the statistical analysis of such 4D surfaces are two-fold. First,
surfaces within the same 4D surface and across different 4D surfaces come with
arbitrary poses and registrations. Second, 4D surfaces can have different
execution rates, e.g., two smiling expressions performed at different speeds.
Thus, to compare and perform statistical analysis on samples of 4D surfaces,
we first need to spatiotemporally register them.

We solve the spatiotemporal registration problem using tools from differential
geometry. We treat surfaces as points in a Riemannian shape space equipped
with an elastic metric that captures shape differences using bending and
stretching energies. We then formulate the elastic registration problem,
_i.e.,_ the problem of computing spatial correspondences, as that of finding
the optimal rotation and reparameterization that align one surface onto
another. This enables comparing and spatially registering surfaces, even in
the presence of large elastic deformations (Section 3.1).

With this representation, a 4D surface becomes a time-parameterized trajectory
in the above-referenced Riemannian shape space. Thus, the problem of analyzing
4D surfaces is reduced to the problem of analyzing curves. Similar to
surfaces, we define a space of curves equipped with a Riemannian metric, which
quantifies the amount of elastic deformation, or time warping, needed to align
two 4D surfaces (Section 3.2).

###  3.1 The elastic shape space of surfaces

Figure 1:  Overview of the proposed spatial registration framework. Surfaces
are first mapped onto the space of Square-Root Normal Fields (SRNF) and
spatially registered using the $\mathbb{L}^{2}$ metric, which is equivalent to
the partial elastic metric in the original space of surfaces. 4D surfaces can
then be treated as curves embedded in the $\mathbb{L}^{2}$ space of SRNFs. The
operator $\star$ refers to the composition of functions in the SRNF space.

Fig. 1 overviews the proposed spatial registration framework. We consider a
surface as a function $f$ of the form:

$\displaystyle f$ $\displaystyle:$ $\displaystyle\Omega\to{}^{3};\hskip
12.0pts\mapsto f(s)=(X(s),Y(s),Z(s)),$ (1)

where $\Omega$ is a parameterization domain and $s\in\Omega$ is the parameter
in this domain. The choice of $\Omega$ depends on the nature of the surfaces
of interest. When dealing with closed surfaces of genus-0, $\Omega$ is a
sphere, _i.e.,_ $\Omega=\mathbb{S}^{2}$, and $s=(u,v)$, with $u\in[0,\pi]$ and
$v\in[0,2\pi[$, are the spherical coordinates. In practice, surfaces come as
unregistered triangulated meshes. We then use the spherical parameterization
algorithm of [40] to map them to a spherical domain.

To remove shape-preserving transformations, we first translate the surfaces so
that their center of mass is located at the origin, and then scale them to
have unit surface area. The space normalized surfaces, denoted by
$\mathcal{F}$, is called the _preshape space_.

Having removed translation and scale, we still need to account for rotations
and reparameterizations. Those are handled algebraically. For any surface
$f\in\mathcal{F}$ and for any rotation $O\in SO(3)$, $Of$ and $f$ have
equivalent shapes. Similarly, any reparameterization of a surface with an
orientation-preserving diffeomorphism preserves its shape. Let $\Gamma$ be the
space of all orientation-preserving diffeomorphisms of $\Omega$. Then,
$\forall\ \gamma\in\Gamma$, $f$ and $f\circ\gamma$, _i.e.,_ the
reparameterization of $f$ with $\gamma$, have the same shape. (Here, $\circ$
refers to the composition of two functions.) Note that reparameterizations
provide dense correspondences across surfaces. If one wants to put a surface
$f_{2}$ in correspondence with another surface $f_{1}$, then we need to find a
rotation $O^{*}$ and a reparameterization $\gamma^{*}$ such that
$O^{*}(f_{2}\circ\gamma^{*})$ is as close as possible to $f_{1}$. This is
precisely the process of 3D surface registration. It is defined mathematically
as:

$(O^{*},\gamma^{*})=\mathop{\rm argmin}_{O\in
SO(3),\gamma\in\Gamma}d_{\mathcal{F}}(f_{1},O(f_{2}\circ\gamma)),$ (2)

where $d_{\mathcal{F}}$ is a measure of distances between surfaces in
$\mathcal{F}$.

####  3.1.1 SRNF representation of surfaces

For efficient registration and comparison of surfaces, the distance measure,
or metric, $d_{\mathcal{F}}$ should quantify interpretable shape differences,
_i.e.,_ the amount of bending and stretching one needs to apply to one surface
to deform it into another. It should also be simple enough to facilitate
efficient computation of correspondences and geodesic paths. Jermyn _et al._
[5] introduced a partial elastic metric that measures differences between
surfaces as a weighted sum of the amount of bending and stretching that one
needs to apply to a surface to align it to another. In this approach, bending
is measured in terms of changes in the orientation of the unit normal vectors,
while stretching is measured in terms of changes in the infinitesimal surface
areas. More importantly, Jermyn _et al._ [5] showed that by using a special
representation of surfaces, called the Square-Root Normal Field (SRNF), the
complex partial elastic metric reduces to the simple $\mathbb{L}^{2}$ metric
on the SRNF space.

######  Definition 3.1 (SRNF maps)

The SRNF map $H(f)$ of a surface $f\in\mathcal{F}$ is defined as the normal
vector field of the surface scaled by the square-root of the local area around
each surface point:

$\displaystyle H:\ $ $\displaystyle\mathcal{F}\rightarrow\mathcal{C}_{h}$
$\displaystyle f\mapsto H(f)=h\text{, such that
}h(u,v)=\frac{\textbf{n}(u,v)}{\|\textbf{n}(u,v)\|^{\frac{1}{2}}_{2}},$ (3)

where $\mathcal{C}_{h}$ is the space of all SRNFs, $\textbf{n}=\frac{\partial
f}{\partial u}\times\frac{\partial f}{\partial v}$ is the normal field to $f$
and $\|\cdot\|_{2}$ is the $\mathbb{L}^{2}$ norm in 3.

The SNRF representation of surfaces has nice properties that make it suitable
for the various analysis tasks at hand:

  * •

It is translation invariant. Also, the SRNF of a rotated surface is simply the
rotation of the SRNF of that surface, _i.e.,_ $H(Of)=OH(f)$.

  * •

$\forall\gamma\in\Gamma$,
$H(f\circ\gamma)=\sqrt{|J_{\gamma}|}(h\circ\gamma)\equiv h\ast\gamma$, where
$J_{\gamma}$ is the Jacobian of $\gamma$ and $|\cdot|$ is its determinant.

  * •

Under the $\mathbb{L}^{2}$ metric in the space of SRNFs, the action of
$\Gamma$ is by isometries, _i.e.,_ $\forall\ \gamma\in\Gamma$ and $\forall\
f_{1},f_{2}\in\mathcal{F},\ \|h_{1}-h_{2}\|=\|h_{1}\ast\gamma-
h_{2}\ast\gamma\|$, where $h_{i}=H(f_{i}),\ i=1,2$.

  * •

The space of SRNFs is a subset of $\mathbb{L}^{2}(\Omega,{}^{3})$. In
addition, the $\mathbb{L}^{2}$ metric in $\mathcal{C}_{h}$ is equivalent to
the partial elastic metric in the space of surfaces. As such, geodesics in
$\mathcal{F}$ become straight lines in the SRNF space $\mathcal{C}_{h}$; see
Fig. 1.

  * •

Currently, there is no analytical expression for the inverse SRNF map, and in
fact, the injectivity and surjectivity of the SRNF remain open questions.
However, Laga _et al._ [4] showed that, for a given SRNF of a valid surface,
one can always numerically estimate the original surface, up to translation
[4].

The last three properties are critical for comparison and atlas construction
of 4D surfaces. One can perform elastic registration of surfaces using the
standard $\mathbb{L}^{2}$ metric in the space of SRNFs, which is
computationally very efficient compared to using the complex elastic metric in
the space of surfaces (Section 3.1.2). Further, temporal evolutions of
surfaces can be interpreted as curves in the Euclidean space of SRNFs, making
them amenable to statistical analysis. Thus, the problem of constructing 4D
atlases becomes the problem of statistical analysis of elastic curves in the
space of SRNFs using standard statistical tools developed for Euclidean
spaces. After analysis, the results can be mapped back to the original space
of surfaces using efficient SRNF inversion procedures [4] (Section 3.2).

####  3.1.2 Spatial elastic registration of surfaces

Under the SRNF representation, the elastic registration problem in Eqn. (2)
can be reformulated using the $\mathbb{L}^{2}$ metric on $\mathcal{C}_{h}$,
the space of SRNFs, instead of the complex partial elastic metric on the
preshape space $\mathcal{F}$. Let $f_{1}$ and $f_{2}$ be two surfaces in the
preshape space $\mathcal{F}$, and $h_{1}$ and $h_{2}$ their SRNFs. Then, the
rotation and reparameterization that optimally register $f_{2}$ to $f_{1}$ are
given by:

$(O^{*},\gamma^{*})=\mathop{\rm argmin}_{O\in
SO(3),\gamma\in\Gamma}\|h_{1}-O(h_{2}\ast\gamma)\|,$ (4)

where $\ast$ is the composition operator between an SRNF and a diffeomorphism
$\gamma\in\Gamma$. This joint optimization over $SO(3)$ and $\Gamma$ can be
solved by alternating, until convergence, between the two marginal
optimizations (this is allowed due to the product structure of
$SO(3)\times\Gamma$) [41]:

  * •

Assuming a fixed parameterization, solve for the optimal rotation using
Procrustes analysis via Singular Value Decomposition (SVD).

  * •

Assuming a fixed rotation, solve for the optimal reparameterization using a
gradient descent algorithm.

To solve for the optimal reparameterization, we represent the space $\Gamma$
of diffeomorphisms $\gamma$, which are functions on the sphere, using
spherical harmonic basis $\\{B_{i}\\}_{i=1,\dots,n}$. This way, every
$\gamma\in\Gamma$ can be written as a weighted sum of the harmonic basis:
$\gamma=\sum_{i=1}^{n}a_{i}B_{i}$. Thus, the search for the optimal
diffeomorphism is reduced to the search for the optimal weights $\\{a_{i}\\}$.
This procedure is described in detail in Section A.1 of the Supplementary
Material.

Although this approach converges to a local optimum, in practice, it can be
used in a very efficient way. Since a 4D surface $\alpha$ is a sequence of
discrete realizations $\alpha(t_{i})\in\mathcal{F},\ i=0,\cdots,n$, with
$t_{0}=0$ and $t_{n}=1$, one can perform the elastic registration
sequentially. Let $\beta=H(\alpha)$ be the SRNF map of the 4D surface
$\alpha$, _i.e.,_ $\forall\ t\in[0,1],\ \beta(t)=H(\alpha(t))$. Also, let
$\alpha_{0}$ be a reference surface randomly chosen from the population of
surfaces being analyzed, and $\beta_{0}$ its SRNF map ($\alpha_{0}$ can be,
for example, $\alpha(0)$). Then,

  1. 1.

Find $O_{0}\in SO(3)$ and $\gamma_{0}\in\Gamma$ that register $\beta(t_{0})$
(the start point of SRNF path) to the SRNF of the reference surface
$\beta_{0}$, by solving Eqn. (4).

  2. 2.

For $i=0,\dots,n$,

     * •

$\beta(t_{i})\leftarrow O_{0}(\beta(t_{i})\ast\gamma_{0}$) and
$\alpha(t_{i})\leftarrow O_{0}\alpha(t_{i})\circ\gamma_{0}$.

  3. 3.

For $i=1,\dots,n$,

     * •

Find, by solving Eqn. (4), $O_{i}\in SO(3)$ and $\gamma_{i}\in\Gamma$ that
register $\beta(t_{i})$ to $\beta(t_{i-1})$.

     * •

$\beta(t_{i})\leftarrow O_{i}(\beta(t_{i})\ast\gamma_{i}$) and
$\alpha(t_{i})\leftarrow O_{i}\alpha(t_{i})\circ\gamma_{i}$.

The first step ensures that, when given a collection of 4D surfaces
$\alpha_{j},\ j=1,\cdots,n$, the surfaces $\alpha_{j}(0),\ j=1,\cdots,n$ are
registered to each other. The subsequent steps ensure that $\forall t,\
\alpha_{j}(t)$ is registered to $\alpha_{j}(0)$. This sequential approach is
efficient since, in general, elastic deformations between two consecutive
frames in a 4D surface are relatively small. In what follows, we assume that
all surfaces within a 4D surface and across 4D surfaces are correctly
registered, _i.e.,_ they have been normalized for translation and scale, and
optimally rotated and reparameterized using the approach described in this
section.

###  3.2 The shape space of 4D surfaces

Figure 2:  In the proposed temporal registration framework, 4D surfaces,
represented as curves in the SRNF space, are first mapped to the space of
Transported Square-Root Vector Fields (TSRVF) for their temporal registration.
Points in the TSRVF space are mapped back to the space of SRNFs and then to
the original space of surfaces for visualization. The operator $\odot$ refers
to the composition of functions in the TSRVF space.

Under the setup of Section 3.1, a 4D surface becomes a curve
$\alpha:[0,1]\to\mathcal{F}$. However, since $\mathcal{F}$ is endowed with the
partial elastic metric, which is non-Euclidean, we propose to further map the
4D surfaces to the SRNF space, which has a Euclidean structure. Thus, 4D
surfaces become curves of the form $\beta:[0,1]\to\mathcal{C}_{h}$. With this
representation, all statistical tasks are carried out in $\mathcal{C}_{h}$
under the $\mathbb{L}^{2}$ metric with results mapped back to the space of
surfaces $\mathcal{F}$ for visualization.

####  3.2.1 TSRVF representation of SRNF trajectories

Let $\alpha$ be a curve (path) in $\mathcal{F}$ and $\beta$ its image under
the SRNF map, _i.e.,_ $\forall\ t\in[0,1],\ \beta(t)=H(\alpha(t))$; $\beta$ is
also a curve, but in $\mathcal{C}_{h}$. Let $\mathcal{M}_{\mathcal{F}}$ be the
space of all paths in $\mathcal{F}$, and $\mathcal{M}_{h}$ be the space of all
paths in $\mathcal{C}_{h}$:
$\mathcal{M}_{h}=\\{\beta:[0,1]\to\mathcal{C}_{h}|\beta=H(\alpha),\
\alpha\in\mathcal{M}_{\mathcal{F}}\\}$.

To temporally register, compare, and summarize samples of such curves, we need
to define an appropriate metric on $\mathcal{M}_{\mathcal{F}}$, or
$\mathcal{M}_{h}$, that is invariant to the rate (or speed) of the 4D
surfaces. For example, facial expressions that only differ in the rate of
their execution should be deemed equivalent under such a metric. Let
$\Xi=\\{\xi:[0,1]\to[0,1]\text{ such that }0<\dot{\xi}<\infty,\xi(0)=0\text{
and }\xi(1)=1\\}$ denote all reparameterizations of the temporal domain
$[0,1]$. Here, $\dot{\xi}=\frac{d\xi}{dt}$. Then, for any $\xi\in\Xi$,
$\beta\circ\xi$ and $\beta$ only differ in the rate of execution and are thus
equivalent. The function $\xi$ is often referred to as a time warping of the
domain $[0,1]$. Temporal registration of two 4D surfaces $\alpha_{1}$ and
$\alpha_{2}$ then becomes the problem of registering their corresponding
curves $\beta_{1}$ and $\beta_{2}$ in $\mathcal{C}_{h}$. This requires solving
for an optimal reparameterization $\xi^{*}\in\Xi$ that minimizes an
appropriate distance $d(\cdot,\cdot)$ between $\beta_{1}$ and $\beta_{2}$:

$\xi^{*}=\mathop{\rm argmin}_{\xi\in\Xi}d(\beta_{1},\beta_{2}\circ\xi).$ (5)

The optimization over $\Xi$ in Eqn. (5) ensures rate invariance. Thus, we are
left with defining a distance $d(\cdot,\cdot)$ that is invariant to time
warping of the temporal domain $[0,1]$. To this end, we borrow tools from
Srivastava _et al._ [36] for analyzing shapes of curves in ${}^{n},n\geq 2$.
The associated elastic metric defined therein is invariant to
reparameterizations of curves, and quantifies the amount of bending and
stretching of the curves in terms of changes in the orientations and lengths
of their tangent vectors, respectively. However, instead of directly working
with such a complex elastic metric, Su _et al._ [42] introduced the
Transported Square-Root Vector Field (TSRVF) representation, which simplifies
the complex elastic metric into the simple $\mathbb{L}^{2}$ metric.

######  Definition 3.2 (Transported Square-Root Vector Field (TSRVF))

For any smooth trajectory $\beta\in\mathcal{M}_{h}$, the transported square-
root vector field (TSRVF) is a parallel transport of a scaled velocity vector
field of $\beta$ to a reference point $c\in\mathcal{C}_{h}$ according to

$Q(\beta)(t)=q(t)=\frac{\dot{\beta}(t)|_{\beta(t)\to
c}}{\sqrt{\|\dot{\beta}(t)\|}},$ (6)

where $\dot{\beta}=\frac{\partial\beta}{\partial t}$ is the tangent vector
field on $\beta$ and $\|\cdot\|$ is the $\mathbb{L}^{2}$ metric on
$\mathcal{C}_{h}$.

Note that the parallel transport $\dot{\beta}(t)|_{\beta(t)\to c}$ is
performed along the geodesic from $\beta(t)$ to $c$. The TSRVF representation
has nice properties that facilitate efficient temporal registration of 4D
surfaces. Let $\beta_{1}$ and $\beta_{2}$ be two trajectories on
$\mathcal{M}_{h}$, and let $q_{1}$ and $q_{2}$ be their respective TSRVFs.
Then,

  * •

The elastic metric on the space of trajectories $\mathcal{M}_{h}$ reduces to
the $\mathbb{L}^{2}$ metric on the space of their TSRVFs. Thus, one can use
the $\mathbb{L}^{2}$ metric to compare two paths:

$d(\beta_{1},\beta_{2})=\|q_{1}-q_{2}\|=\left(\int_{0}^{1}\|q_{1}(t)-q_{2}(t)\|^{2}dt\right)^{\frac{1}{2}},$
(7)

where $\|\cdot\|$ is again the $\mathbb{L}^{2}$ norm on $\mathcal{C}_{h}$.

  * •

For any $\xi\in\Xi$, $Q(\beta\circ\xi)=(q\circ\xi)\sqrt{\dot{\xi}(t)}$, which
we denote by $q\odot\xi$.

  * •

Under the $\mathbb{L}^{2}$ metric, the action of the reparameterization group
$\Xi$ on the space of TSRVFs is by isometries, _i.e.,_
$\|q_{1}-q_{2}\|=\|(q_{1}\odot\xi)-(q_{2}\odot\xi)\|,\ \forall\ \xi\in\Xi$.

  * •

Given a TSRVF $q$ and an initial trajectory point, one can reconstruct the
corresponding path $\beta$, such that $Q(\beta)=q$, by solving an ordinary
differential equation [42].

As we will see next, these properties enable efficient temporal registration
of trajectories and subsequent rate-invariant statistical analysis. In what
follows, let $\mathcal{Q}$ denote the space of TSRVFs equipped with the
$\mathbb{L}^{2}$ metric defined in Eqn. 7.

####  3.2.2 Temporal registration

Under the TSRVF representation, the temporal registration problem in Eqn. (5),
which involved optimization over $\Xi$, can now be reformulated using the
standard $\mathbb{L}^{2}$ metric on the space of TSRVFs:

$\xi^{*}=\mathop{\rm argmin}_{\xi\in\Xi}\|q_{1}-q_{2}\odot\xi\|.$ (8)

This problem can be solved efficiently using a Dynamic Programming algorithm
[43, 42]. Then, the rate-invariant distance between two trajectories is given
by:

$d(\beta_{1},\beta_{2})=\inf_{\xi\in\Xi}\|q_{1}-q_{2}\odot\xi\|.$ (9)

Figure 3:  Example of a geodesic between the source 4D surface (top row) and
the target 4D surface (bottom row) after spatiotemporal registration. The
highlighted row corresponds to the mean 4D surface. A video of the figure is
included in the Supplementary Material.

####  3.2.3 Geodesics between 4D surfaces

We now summarize our entire pipeline. Let
$\alpha_{1},\alpha_{2}\in\mathcal{M}_{\mathcal{F}}$ be two 4D surfaces. The
pipeline to spatiotemporally register them and compute the geodesic path
between them can be summarized as follows.

(1) Proposed spatial registration.  The goal is to spatially register the
surfaces in $\alpha_{1}$ and $\alpha_{2}$ to the same reference surface, which
can be any arbitrary surface. For simplicity, we choose it to be
$\alpha_{1}(0)$, the first surface in the sequence $\alpha_{1}$. The spatial
registration can then be performed in two steps.

  * •

Compute the SRNF maps, $\forall\ t\in[0,1]$, $\beta_{1}(t)=H(\alpha_{1}(t))$
and $\beta_{2}(t)=H(\alpha_{2}(t))$.

  * •

Spatially register $\beta_{1}$ and $\beta_{2}$, and thus $\alpha_{1}$ and
$\alpha_{2}$, to the reference surface, using the algorithm described in
Section 3.1.2.

For simplicity of notation, we also use $\beta_{1}$ and $\beta_{2}$ to denote
the spatially-registered trajectories.

(2) Proposed temporal alignment. $\beta_{1}$ and $\beta_{2}$ are elements of
$\mathcal{M}_{h}$. We perform temporal registration in three steps.

  * •

Map $\beta_{1}$ and $\beta_{2}$ to the TSRVF space
$\mathcal{Q}:q_{1}=Q(\beta_{1})$ and $q_{2}=Q(\beta_{2})$.

  * •

Find $\xi^{*}$, the optimal reparametrization that registers $q_{2}$ to
$q_{1}$ by solving Eqn. (8).

  * •

$q_{2}^{*}\leftarrow q_{2}\odot\xi^{*}$ and
$\beta_{2}^{*}\leftarrow\beta_{2}\circ\xi^{*}$.

(3) Proposed geodesic computation.  Since $\mathcal{Q}$ is Euclidean, the
geodesic path $\Lambda_{q}$ between $q_{1}$ and $q_{2}^{*}$ is a straight
line:

$\Lambda_{q}(\tau)=(1-\tau)q_{1}+\tau q_{2}^{*},\ \tau\in[0,1].$ (10)

Next, we map $\Lambda_{q}$ back to $\mathcal{M}_{h}$ using the inverse TSRVF
map, _i.e.,_ $\forall\ \tau,\
\Lambda_{\beta}(\tau)=Q^{-1}(\Lambda_{q}(\tau))$. The computation of the
inverse mapping uses the starting point on the trajectory and has a closed-
form solution, making it computationally efficient. This is described in
detail in [42]. After applying the inverse mapping to the entire geodesic
path, we have $\Lambda_{\beta}(0)=\beta_{1},\ \Lambda_{\beta}(1)=\beta_{2}$,
and $\beta_{\tau}=\Lambda_{\beta}(\tau),\ \tau\in(0,1)$, _i.e.,_ a geodesic
path between the SRNF curves $\beta_{1}$ and $\beta_{2}$.

(4) Visualization.  To visualize geodesic paths between 4D surfaces (and not
their SRNFs), we need to further map all SRNFs on the trajectory
$\Lambda_{\beta}(\tau)$ to their corresponding surfaces in $\mathcal{F}$. This
is done using the inverse SRNF map, _i.e.,_ $\forall\ \tau\in[0,1],\
t\in[0,1],\ \Lambda(\tau)(t)=H^{-1}(\Lambda_{\beta}(\tau)(t)).$ Unlike the
TSRVF map whose inverse can be computed analytically, inversion of the SRNF
map, whose injectivity and surjectivity are yet to be determined, has to be
accomplished numerically using the approach of Laga _et al._ [4].

Now, $\Lambda$ is the geodesic path between the 4D surfaces $\alpha_{1}$ and
$\alpha_{2}^{*}$, _i.e.,_ $\Lambda(0)=\alpha_{1},\Lambda(1)=\alpha_{2}^{*}$,
and $\alpha_{\tau}=\Lambda(\tau)$ is a 4D surface at time $\tau$ along the
geodesic path. Fig. 3 shows an example of a geodesic between two 4D surfaces
representing talking faces. Each row corresponds to one 4D surface. The top is
the source, the bottom row is the target, after optimal spatiotemporal
registration, and the highlighted row in the middle corresponds to the mean 4D
surface. The temporal registration is further illustrated in Fig. 4, where we
show the source 4D surface, the target 4D surface before the spatiotemporal
registration, and the target 4D surface after the spatiotemporal registration.
Section 5 provides more examples of geodesics computed between various types
of 4D surfaces.

##  4 Statistical analysis of 4D surfaces

Now that we have devised all of the required mathematical tools for comparing
4D surfaces, we shift our focus to how these tools can be used to build a 4D
atlas from a sample of 4D surfaces. Let $\alpha_{1},\cdots,\alpha_{n}$ be a
set of 4D surfaces and $\beta_{1},\cdots,\beta_{n}$ be their corresponding
trajectories in $\mathcal{C}_{h}$. We assume that all of the surfaces, and
their corresponding SRNFs, have been spatially registered to a common
reference; see Section 3.1.2. We proceed to map all of the 4D surfaces to
their corresponding TSRVFs, hereinafter denoted by $q_{1},\cdots,q_{n}$, and
compute statistics in the space of TSRVFs. As before, all results are mapped
at the end to the original space of surfaces $\mathcal{F}$ for visualization.
We will use this framework to compute means and modes of variation, and to
synthesize novel 4D surfaces by sampling from probability distributions fitted
to a set of exemplar 4D surfaces.

Mean of 4D surfaces.  Intuitively, the mean of a collection of 4D surfaces is
the 4D surface that is as close as possible to all of the 4D surfaces in the
collection, under the specified distance measure (or metric). It is also
called Karcher mean and is defined as the 4D surface that minimizes the sum of
squared distances to all of the 4D surfaces in the given sample. In other
words, we seek to solve the following optimization problem, defined in the
space of TSRVFs:

$\bar{q}=\mathop{\rm
argmin}_{q\in\mathcal{Q}}\sum_{i=1}^{n}\min_{\xi_{i}\in\Xi}\|q-q_{i}\odot\xi_{i}\|^{2}.$
(11)

Algorithm 3 in the Supplementary Material describes the proposed procedure for
solving this optimisation problem. It outputs the TSRVF Karcher mean
$\bar{q}$, the optimal temporal reparameterizations $\xi_{i}^{*},\
i=1,\ldots,n$, and the temporally registered TSRVFs
$q_{i}^{*}=q_{i}\odot\xi_{i}^{*}$; again, for simplified notation we simply
use $\xi_{i}$ and $q_{i}$ to denote the optimal temporal reparameterizations
and the temporally registered TSRVFs. The mean 4D surface can be obtained by
TSRVF inversion of the mean TSRVF followed by SRNF inversion [4].

---
Figure 4:  Examples of the spatiotemporal registration of two facial
expressions (4D faces). In each example, we show (a) the source 4D face, (b)
the target 4D face, and (c) the target 4D face after spatiotemporal
registration using the proposed framework. Note how the spatiotemporally
registered target 4D surface became fully synchronised with the source 4D
surface. The full video sequence is provided in the supplementary material.

Principal directions of variation. Since the TSRVF space is Euclidean, the
principal directions of variation can also be computed in a standard way,
_i.e.,_ using the Singular Value Decomposition (SVD) of the covariance matrix.
In the following, we assume that the TSRVFs are sampled using a finite set of
points and appropriately vectorized. Let
$K=\frac{1}{n-1}(q_{i}-\bar{q})(q_{i}-\bar{q})^{\top}$ be the covariance
matrix of the input sample, $\sigma_{i},\ i=1,\dots,k$ its $k-$leading
eigenvalues, and $\Sigma_{i},\ i=1,\dots,k$ the corresponding eigenvectors.
Then, one can explore the variability in the $i-$th principal direction using
$q_{\tau}=\bar{q}+\tau\sqrt{\sigma_{i}}\Sigma_{i}$, where $\tau\in\real$. To
visualize this principal direction of variation, we again use TSRVF inversion
followed by SRNF inversion to compute the 4D surface $\alpha_{\tau}$, such
that $Q(H(\alpha_{\tau}))=\bar{q}+\tau\sqrt{\sigma_{i}}\Sigma_{i},\
\tau\in\real$.

Random 4D surface synthesis.  Given the mean and the $k$-leading principal
directions of variation, any TSRVF $q$ of a 4D surface $\alpha$ can be
approximately represented, in a parameterized form, as:

$q=\bar{q}+\sum_{i=1}^{k}\tau_{i}\sqrt{\sigma_{i}}\Sigma_{i},\tau_{i}\in\real.$
(12)

Thus, to generate a random TSRVF, we only need to generate $k$ random values
$\tau_{i}\in\real$ and plug them into Eqn. (12). Then, to compute the
corresponding random 4D surface, we apply the inverse TSRVF map followed by
the inverse SRNF map. Also, by enforcing each $\tau_{i}$ to be within a
certain range, e.g., $[-1,1]$, we can ensure that the generated random 4D
surfaces are similar to the given sample and thus plausible.

This procedure allows the generation of new random 4D surfaces. However, it
does not offer any control over the generation process, which is entirely
random. In many situations, we would like to control this process using a set
of parameters. For instance, when dealing with 4D facial expressions, these
parameters can be the degree of sadness, facial dimensions, etc. This type of
control can be implemented using regression in the TSRVF space, a problem that
we plan to explore in the future.

Figure 5:  Example of the spatiotemporal registration, using the proposed
algorithm, of two 4D human body shapes (from the DFAUST dataset) performing a
jumping action at different speeds. Note how the spatiotemporally registered
target 4D surface in (c) became synchronised with the source 4D surface in
(a). The full video sequence is provided in the supplementary material.

##  5 Results

This section demonstrates some results of the proposed framework and evaluates
its performance. Section 5.1 focuses on spatiotemporal registration and
geodesic computation between 4D surfaces. Section 5.2 focuses on the
computation of the statistical summaries while Section 5.3 focuses on the
random synthesis of 4D surfaces. Finally, Section 5.4 provides an ablation
study to demonstrate the importance of each component of the proposed
framework. We use three datasets: (1) VOCA [6], which contains 4D facial
scans, captured at $60$fps, of $12$ subjects speaking various sentences; (2)
MPI DFAUST [7], which contains high-resolution 4D scans of $10$ human subjects
in motion, captured at $60$fps; in total, the dataset contains $129$ dynamic
performances; and (3) MPI 4D CAPE [8], which contains high-resolution 4D scans
of $10$ male and $5$ female subjects in clothing. These datasets come as
polygonal meshes with consistent triangulation and given registration across
the meshes. We spherically parameterize them using Kurtek _et al._ ’s
implementation [17] of the spherical parameterization approach of [40]. We
also apply randomly generated spatial diffeomorphisms to simulate non-
registered surfaces. Our framework does not use, either explicitly or
implicitly, the provided vertex-wise correspondences.

Figure 6:  Example of the spatiotemporal registration, using the proposed
algorithm, of two 4D body shapes with different clothing ( from the CAPE
dataset). Note how the spatiotemporally registered target 4D surface in (c)
became fully synchronised with the source 4D surface in (a). The full video
sequence is provided in the supplementary material.
---
(a) Before registration. The highlighted middle row corresponds to the mean 4D
surface.
(b) After registration. The highlighted middle row corresponds to the mean 4D
surface.
Figure 7:  Example of a geodesic between 4D surfaces corresponding to punching
actions ((a) before registration and (b) after registration). In each example,
we show the source 4D surface in the first row, the target 4D surface in the
last row, and three intermediate 4D surfaces along the geodesic between the
source and the target. Observe how misaligned are the highlighted frames
before registration. A video illustrating these sequences is included in the
Supplementary Material.

###  5.1 Spatiotemporal registration and 4D geodesics

We consider pairs of 4D facial expressions from the VOCA dataset. We first
reparameterize each 4D surface using randomly generated time-warping functions
to simulate facial expressions performed at different execution rates. We then
apply the framework proposed in this paper to spatiotemporally register them.
Fig. 4 shows an example of such spatiotemporal registration. In this example,
we show (a) the source 4D surface, (b) the target 4D surface before
spatiotemporal registration, and (c) the target 4D surface after
spatiotemporally registering it to the source. We also highlight some key
frames. As one can see, the original 4D surfaces differ significantly in their
execution rates. The spatiotemporal registration framework synchronizes the
source and target expressions, thus enabling their comparison, interpolation
and averaging. We also perform a similar experiment on the human body shapes
of the DFAUST [7] and CAPE [8] datasets; see Figs. 5, 6, and 12(a)-(c).
Compared to faces, human body shapes are very challenging to analyze since
they perform complex articulated motions, which result in large bending and
stretching of their surfaces.

4D geodesics.  Fig. 7 shows geodesics between 4D human body shapes. In this
example, both the source and target perform a punching action but at different
rates. We show the geodesic before and after the spatiotemporal registration
of the target 4D surface onto the source. Unlike the jumping action in Fig. 5,
the left hand of the target surface does not perform the same action as the
left hand of the source surface. Nevertheless, our framework can bring these
two 4D surfaces as close as possible to each other. The supplementary material
includes a video of the sequence and more examples of geodesics between 4D
faces (from VOCA), 4D human bodies (from DFAUST), and clothed 4D human bodies
(from CAPE).

TABLE I: Comparison of the performance and accuracy of the proposed spatial registration with state-of-the-art techniques such as MAP Tree [44] and Fast Sinkhorn filters [45], which are based on functional maps [27].  |  COMA [46] |  CAPE [8] |  DFAUST [7]
---|---|---|---
| Mean | Std | Median | Mean | Std | Median | Mean | Std | Median
MapTree [44] | $1.6042$ | $0.6956$ | $1.6069$ | $1.4447$ | $0.7227$ | $1.4483$ | $1.5344$ | $0.7065$ | $1.5211$
ICP-NN [45] | $1.6028$ | $0.6973$ | $1.6056$ | $1.4684$ | $0.7087$ | $1.5116$ | $1.5185$ | $0.6905$ | $1.5082$
ICP-Sinkhorn [45] | $1.6019$ | $0.6974$ | $1.6053$ | $1.4684$ | $0.7037$ | $1.5058$ | $1.5275$ | $0.6888$ | $1.5227$
Zoomout-NN [45] | $1.5997$ | $0.6902$ | $1.6002$ | $1.4743$ | $0.7029$ | $1.4781$ | $1.5101$ | $0.6922$ | $1.4857$
Zoomout-Sinkhorn [45] | $1.6016$ | $0.6908$ | $1.5968$ | $1.4737$ | $0.6937$ | $1.4925$ | $1.5019$ | $0.6948$ | $1.4731$
SRNF (ours) | 0.0012 | 0.0008 | 0.0003 | 0.0008 | 0.0008 | 0.0006 | 0.0008 | 0.0008 | 0.0007

Evaluation of the spatial registration. We quantitatively evaluate the
accuracy of the proposed spatial registration method and compare it to the
latest functional map-based techniques such as MapTree [44] and Fast Sinkhorn
filters [45]. Similar to our method, functional maps operate on clean manifold
surfaces and do not use any form of (deep) learning. We take the surfaces of
COMA [46], CAPE [8], and DFAUST [7] datasets, which come with ground-truth
correspondences, and apply random spatial diffeomorphisms to them to simulate
unregistered surfaces. We then compute the correspondence map between each
pair of surfaces in the dataset. We measure the spatial registration error in
terms of the geodesic distance, on the parameterization domain, between the
ground-truth and the computed correspondence. Table I reports the mean,
standard deviation, and median of the registration errors computed across all
the models in each data set. As one can see, the proposed SRNF-based spatial
registration method significantly outperforms state-of-the-art algorithms [44,
45]. We refer the reader to the supplementary material, which includes visual
examples of pairs of surfaces before and after spatial registration. It also
includes additional spatial registration experiments using the quadruped
animal data set of Kulkarni _et al._ [47].

An important property of the proposed approach is that it finds a one-to-one
mapping between the source and target surfaces. This is not the case with
functional map-based methods, which can map a point on the source to multiple
points on the target. Thus, they cannot be used to compute geodesics,
interpolations, and statistical summaries.

Evaluation of the temporal registration. We use the FLAME fitting framework
[31] to generate random 4D facial surfaces with known ground-truth temporal
registrations. We first generate two random SMPL parameters, each
corresponding to a 3D surface, and then linearly interpolate them to simulate
a deforming 4D facial surface. Let $\alpha_{i},\ i\in\\{1,\dots,100\\}$ be the
resulting 4D surfaces. Next, we generate $100$ random temporal diffeomorphisms
$\xi_{i}$; see Fig. 24-(b) of the supplementary material. These will be used
to simulate 4D facial surfaces that have different execution rates.

|
---|---
(a) Before registration. | (b) After registration.
Figure 8:  Boxplots of errors between $20$ pairs of 4D surfaces: (a)
unregistered 4D surfaces generated using $100$ random diffeomorphic
transformations of a single 4D surface for $5$ sequences and (b)
spatiotemporally registered surfaces. The red lines represent the media error
and the boxes represent its spread. The green curve in (a) and (b) is the
distance between the perfectly registered 4D surfaces. Figure 9:  Co-
registration of multiple 4D surfaces. In this example, we consider four human
body shapes performing a jumping action (first four rows) and two others
performing a punching action (rows 5 and 6). Here, we show the
spatiotemporally co-registered 4D surfaces and the 4D mean computed using the
proposed algorithm. The supplementary material includes the input 4D surfaces
before their spatiotemporal registration. It also includes the full video
sequences. The surfaces are from the DFAUST dataset.

Now, given a pair of 4D surfaces $\alpha_{i}$ and $\alpha_{j}$, and for each
pair of temporal diffeomosphims $\xi_{k}$ and $\xi_{l}$,
$\alpha_{i}\circ\xi_{k}$ and $\alpha_{j}\circ\xi_{l}$ can be seen as a pair of
4D surfaces with different execution rates. Next, we compute, using the
proposed framework, the optimal diffeomorphism $\xi^{*}_{i,j}$ that aligns
$\alpha_{i}\circ\xi_{k}$ onto $\alpha_{j}\circ\xi_{l}$. Let
$\alpha^{*}_{i}=\alpha_{i}\circ\xi^{*}_{i,j}\circ\xi_{k}$. To quantitatively
evaluate the quality of the computed temporal registration, we compute:

  * •

The distance between the perfectly registered 4D surfaces $\alpha_{i}$ and
$\alpha_{j}$; see the green curves in Fig. 8.

  * •

The distance between $\alpha_{i}\circ\xi_{k}$ and $\alpha_{j}\circ\xi_{l}$
before temporal registration (Fig. 8-(a)), and the distance between
$\alpha^{*}_{i}$ and $\alpha_{j}$, _i.e.,_ the distance between the two 4D
surfaces after their temporal registration (Fig. 8-(b)). Ideally, the latter
should be significantly smaller than the former. It should also be as close as
possible to the distance between the perfectly registered 4D surfaces
$\alpha_{i}$ and $\alpha_{j}$.

Figs. 8-(a) and (b) report statistics of these errors for each pair of 4D
surfaces, but aggregated over the $100$ random diffeomorphisms. As one can
see, the median distance between the 4D surfaces after registration (Fig.
8-(b)) is significantly lower than the one before registration (Fig. 8-(a)).
The former is significantly closer to the ground-truth (shown with green
curves in Fig. 8) than the latter.

Computation time. Our approach runs entirely on a CPU. The Matlab
implementation of the spatiotemporal registration process takes less than
$31.43$ seconds on $4.2$ GHz Intel Core i7 with $32$ GB of RAM. The
visualization, which is needed when computing geodesics, means, and directions
of variation, and when synthesizing random 4D surfaces, relies on the
inversion of the SRNF maps. It requires $6$ seconds per frame and a total of
$30$ minutes for the $300$ temporal frames used in this paper. All the
experiments were performed using a high spherical resolution ($256\times
256$).

---
(a) First mode of variation.
(b) Second mode of variation.
(c) Five randomly synthesized 4D faces.
Figure 10:  First (a) and second (b) principal directions of variation (the
mean 4D surface is highlighted in the middle). Each row corresponds to one 4D
surface sampled between $-1.5$ to $1.5$ times the standard deviation along the
principal direction of variation. We refer the reader to the supplementary
material, which shows the input 4D faces (before their spatiotemporal
registration). It also includes more modes of variation and random samples, as
well as the complete video sequences.

###  5.2 Summary statistics

We now consider a set of unregistered 4D surfaces and compute their mean and
principal directions of variation. Fig. 9 shows the 4D mean (highlighted with
a blue box) computed from six 4D human shapes performing different types of
actions. The figure also shows the input 4D surfaces after their
spatiotemporal registration; see the video in the supplementary material for
an illustration of the input 4D surfaces before spatiotemporal registration.
Despite the large articulated motion, the large differences in the type of
actions and the significant differences in the execution rates of the 4D
surfaces, our framework is able to co-register them and generate a plausible
average 4D surface. Figs. 10-(a) and (b) show the mean and the first two
principal modes of variation computed on input 4D facial surfaces. As we can
see, the computed mean also captures the main features of the dataset. The
principal directions of variation further capture relevant variability in the
given data. The supplementary material includes the input 4D surfaces prior to
their registration. Please also refer to the videos in the supplementary
material for additional results.

###  5.3 4D surface synthesis

Fig. 10-(c) shows five 4D facial expressions randomly sampled from a Gaussian
distribution with parameters estimated from the VOCA dataset using the method
of Sec. 4. To ensure that the synthesized 4D surfaces are plausible, we only
consider those that are within $1.5$ standard deviations along each principal
direction of variation. We refer the reader to the supplementary material for
videos of all of the randomly generated 4D surfaces. The ability to synthesize
novel 4D surfaces can benefit many applications in computer vision and
graphics. It can be used to augment datasets for efficient training of deep
learning models.

###  5.4 Ablation study

We undertake an ablation study to demonstrate the importance of each component
of the proposed framework.

Importance of the SRNF representation. In this experiment (Fig. 11), we take
two challenging 3D human body models, which undergo a large articulated
motion, perform their spatial registration using the proposed SRNF approach,
and then compute their statistical mean using the $\mathbb{L}^{2}$ metric in
the original surface space (Fig. 11-(a)) and the $\mathbb{L}^{2}$ metric in
the SRNF space (Fig. 11-(b)). Fig. 11-(a) shows that the articulated parts of
the mean computed in the original surface space unnaturally shrink. This is
predictable since, under the $\mathbb{L}^{2}$ metric, geodesics correspond to
straight lines. However, in the SRNF space, the $\mathbb{L}^{2}$ metric is
equivalent to the optimal bending and stretching of the surfaces, and thus the
computed mean is more natural; see Fig. 11-(b).

|
---|---
(a) Without SRNF. | (b) With SRNF.
Figure 11:  The mean shape between the left and right surfaces, computed (a)
in the original surface space without the SRNF representation, and (b) in the
SRNF space. In (a), the mean shape is distorted due to the use of the
$\mathbb{L}^{2}$ metric in the originals pace of surfaces. In both cases, the
spatial registration is performed using the proposed registration method.

Next, we consider two full 4D surfaces of deforming human body shapes (Fig.
12-(a) and (b)) and show their mean 4D surface obtained: (1) with the SRNF
representation, with spatial registration, and without temporal registration
(Fig. 12-(d)), (2) with the SRNF representation, with spatial registration,
and with temporal registration (Fig. 12-(e)), (3) without SRNF representation,
with spatial registration, and without temporal registration (Fig. 12-(f)),
and (4) without SRNF representation, with spatial registration, and with
temporal registration (Fig. 12-(g)). The last two cases are equivalent to a
linear interpolation in the original surface space, after spatial
registration. In all cases, we perform the spatial registration using the SRNF
framework.

Figure 12:  Ablation study: illustration of the effect of the different
components of the proposed framework on the quality of the computed mean 4D
surface, which is the middle point along the geodesic between the source and
target 4D surfaces. Te bottom row is a zoom on the frame highlighted in (a) to
(g). The 4D surfaces are from the DFAUST dataset. A video illustrating these
sequences is included in the Supplementary Material.

First, we can see that the temporally-aligned target 4D surface (Fig. 12-(c))
is very close to the source 4D surface in Fig. 12-(a). We observe that the
right hands became fully synchronized. As such, the mean 4D surface obtained
after temporal registration (Fig. 12-(e)) is fully synchronised with the
source and the aligned target, unlike the mean 4D surface in Fig. 12-(d),
which has been obtained without temporal registration. Second, in the mean 4D
surfaces obtained without the SRNF framework (Figs. 12-(f) and (g)), we can
observe that the parts that undergo large articulated motion (e.g., the arms)
unnaturally shrink. This shrinkage is stronger in Fig. 12-(f) since the mean
is obtained without temporal registration. The bottom row of Fig. 12 shows a
zoom on the time frame highlighted in Figs. 12-(a) to (g).

Finally, we quantitatively evaluate the importance of the SRNF representation
by comparing the expressive power of PCA on the original space of spatially
registered surfaces and on the space of SRNFs. We randomly divide a data set
equally into a training set and a testing set. We then fit a PCA model to the
training set (both in the original space and in the space of SRNFs), project
each model in the test set onto the PCA model, reconstruct it, and measure the
error between the original and the reconstructed models. We perform 5-fold
cross-validation. Table II reports the mean, the median, and the standard
deviation of the error over the test set and averaged over the five runs. As
one can see, PCA on the SRNF space has a significantly lower reconstruction
error than PCA in the original space of surfaces. This demonstrates that the
former is more suitable to characterize variability in the shape of 3D objects
that bend and stretch.

TABLE II: Comparison of the expressive power of PCA on the original space of surfaces and on the space of SRNFs. The lower the values are, the better. | PCA on surfaces | PCA on SRNFs
---|---|---
| Mean | Std | Median | Mean | Std | Median
DFAUST | $0.26$ | $0.042$ | $0.270$ | 0.12 | $0.023$ | 0.121
VOCA | $0.030$ | $0.011$ | $0.028$ | 0.009 | $0.004$ | 0.008
CAPE | $0.105$ | $0.041$ | $0.101$ | 0.053 | $0.021$ | 0.050
TABLE III: Comparison of the expressive power of PCA on the original space of curves and on the space of TSRVFs. The lower the values are, the better. | PCA on curves | PCA on TSRVFs
---|---|---
| Mean | Std | Median | Mean | Std | Median
DFAUST | $0.926$ | $1.110$ | $0.693$ | 0.756 | $0.104$ | 0.715
VOCA | $0.676$ | $0.182$ | $0.64$ | 0.486 | $0.318$ | 0.603

Importance of the TSRVF representation for 4D surfaces. We perform a similar
ablation study, but on 4D surfaces, to compare the expressive power of PCA on
the original space of curves and on the space of TSRVFs. Table III shows that
PCA error on the TSRVF space is lower than the error in the original space.
This demonstrates that the former is more suitable to characterize variability
in 4D surfaces.

##  6 Conclusion

We have proposed a new framework for the statistical analysis of longitudinal
3D shape data (or 4D surfaces), _i.e.,_ surfaces that deform over time, e.g.,
3D human body shapes performing actions at different execution rates or 3D
human faces pronouncing sentences at different speeds. Unlike traditional
techniques, which only consider how features such as landmarks or measurements
vary over time, the proposed framework considers the deformation of the entire
surface of a 3D object. Our key contribution is in representing 4D surfaces as
trajectories in the space of SRNFs, and the use of Transported Square-Root
Vector Fields to analyze such trajectories statistically. The proposed
framework can spatiotemporally register 4D surfaces, even in the presence of
large elastic deformations and significant variations in the execution rates.
It is also able to compute geodesics and summary statistics, which in turn can
be used to synthesize new, unseen 4D surfaces randomly.

Although we have demonstrated the proposed 4D analysis framework on human body
shapes and facial surfaces, it is general and can be applied to other types of
surfaces. Our current implementation is limited to surfaces that are
homeomorphic to a sphere, but we plan to extend the framework to higher-genus
surfaces by exploring different parameterization methods, including mesh-based
representations [48]. The approach uses the numerical SRNF inversion procedure
of Laga _et al._ [4], which is sometimes not accurate near the poles of the
parameterization domain; we plan to improve its performance via the use of
charts.

The framework deals with surfaces that bend and stretch but do not change in
topology; as such, it does not apply to tree-like shapes, e.g., botanical
trees or roots. However, the concept of representing deformations as
trajectories in a shape space also applies to tree-shape spaces such as those
used in [49, 50]. The framework is also limited to clean surfaces that are
free of geometric and topological noise; as such, the proposed spatial
registration method cannot be used to register partial scans to each other, or
to register a template to partial scans. However, similar to statistical shape
models such as 3D morphable models and SMPL, the proposed 4D atlas can be used
as a prior; in conjunction with a data generation model, it can thereby be
applied to noisy or partial data, e.g., to reconstruct entire 4D surfaces. The
statistical analysis presented in this paper assumes that the population of
the 4D surfaces follows a Gaussian distribution. We plan to extend the
approach to other types of distributions, e.g., Gaussian Mixture Models, which
can represent populations that follow multimodal distributions

The proposed framework has various applications in computer vision, graphics,
biology, and medicine. In computer vision, collecting large animations to
train deep neural networks, e.g., for 3D reconstruction or action recognition
[51, 52], is complex and time-consuming. Our framework can contribute to
solving this problem by automatically synthesizing new samples from a small
dataset. Our current implementation has only considered random synthesis,
which is very important for populating virtual environments and for data
augmentation to train deep learning networks. However, there are many
situations where we would like to control this process using a set of
parameters. For instance, when dealing with 4D facial expressions, these
parameters can be the degree of sadness, facial dimensions, etc. This type of
control can be implemented efficiently using regression in the TSRVF space.
Finally, our framework can be used to statistically analyze how anatomical
organs deform due to growth or disease progression.

Acknowledgement.  We would like to thank the authors of [12, 7, 46, 6, 8] for
making their datasets publicly available, and [44] and [45] for sharing the
codes of MapTree and the Fast Sinkhorn Filters-based surface registration.

##  Appendix A Spatial registration

This appendix discusses the implementation details of the spatial registration
and provides additional results on various complex 3D shapes.

###  A.1 Algorithm

In this section, we provide more details on the procedure used to solve the
spatial registration problem in Eqn. (4) of the main manuscript. In our
formulation, we consider spatial registration of a surface $f_{2}$ to a
surface $f_{1}$ as the problem of finding the rotation and reparameterization
that bring the SRNF $h_{2}$ of $f_{2}$ as close as possible to the SRNF
$h_{1}$ of $f_{1}$. We measure closeness using the $\mathbb{L}^{2}$ metric in
the SRNF space, which is equivalent to the partial elastic metric in the
original space of surfaces. That is, we seek to solve the following
optimization problem:

$(O^{*},\gamma^{*})=\mathop{\rm argmin}_{O\in
SO(3),\gamma\in\Gamma}\|h_{1}-O(h_{2}\ast\gamma)\|,$ (13)

where $\ast$ is the composition operator between an SRNF and a diffeomorphism
$\gamma\in\Gamma$. This joint optimization over $SO(3)$ and $\Gamma$ can be
solved by alternating, until convergence, between the two marginal
optimizations:

  * •

Assuming a fixed parameterization, solve for the optimal rotation using
Procrustes analysis via Singular Value Decomposition (SVD).

  * •

Assuming a fixed rotation, solve for the optimal reparameterization using a
gradient descent algorithm.

Below we detail each of these steps.

Optimization over the rotation group. For a fixed $\gamma\in\Gamma$, the
minimization over $SO(3)$ can be performed directly using Procrustes analysis.
Let $\tilde{h}_{2}$ denote
$(h_{2},\gamma)\equiv\sqrt{J[\gamma]}\>(h_{2}\circ\gamma)$ in Eqn. (13).
(Here, $J[\gamma]$ is the determinant of the Jacobian of $\gamma$.) The
optimal rotation matrix

$O^{*}=\mathop{\rm argmin}_{O\in SO(3)}\lVert h_{1}-O\tilde{h}_{2}\rVert^{2}$
(14)

can then be obtained using Algorithm 1.

Input: Two surfaces $\\{f_{1},f_{2}\\}\in{\mathcal{F}}$.
Output: Optimal rotation matrix $O^{*}$ and optimally rotated surface
$f_{2}^{*}$.

1: Compute the SRNFs $h_{1}=H(f_{1})$ and $h_{2}=H(f_{2})$.

2: Compute the $3\times 3$ matrix
$A=\int_{\Omega}h_{1}(s){\tilde{h}}_{2}(s)^{\top}ds$.

3: Compute the singular value decomposition $A=U\Sigma V^{T}$.

4: Compute the optimal rotation as $O^{*}=UV^{\top}$. (If the determinant of
$A$ is negative, the last column of $V$ changes sign.)

5: Compute the optimally rotated surface $f_{2}^{*}=O^{*}f_{2}$.

Algorithm 1 Optimal rotational alignment of two surfaces.

Optimization over the space of spatial diffeomorphisms. We use a gradient
descent approach to solve the optimization problem over $\Gamma$. Although
this approach has an obvious limitation of converging to a local solution, it
is still general enough to be applicable to general surfaces and provides
plausible results. In order to specify the gradient, we focus on the current
iteration, and define the reduced cost function
$E_{\text{reg}}:\Gamma\to{}_{\geq 0}$:

$E_{\text{reg}}(\gamma)=\lVert h_{1}-(\tilde{h}_{2},\gamma)\rVert^{2}=\lVert
h_{1}-\phi(\gamma)\rVert^{2},$ (15)

where $\tilde{h}_{2}=(h_{2},\gamma_{0})$, $\gamma_{0}$ and $\gamma$ denote the
current and the incremental reparameterizations respectively, and
$\phi:\Gamma\to[h_{2}]$ is defined to be
$\phi(\gamma)=(\tilde{h}_{2},\gamma)$. Let $b$ be a unit vector in
$T_{\gamma_{\text{id}}}(\Gamma)$ for $\gamma_{\text{id}}(s)=s$. Then, the
directional derivative of $E_{\text{reg}}$ at $\gamma_{\text{id}}$, in the
direction of $b$, is given by $\langle
h_{1}-\phi(\gamma_{\text{id}}),d\phi(b)\rangle b$, where $\phi_{\ast}$ is the
differential of $\phi$ and $\langle\cdot,\cdot\rangle$ is the $\mathbb{L}^{2}$
inner product. If we have an orthonormal basis for
$T_{\gamma_{\text{id}}}(\Gamma)$, we can specify the full gradient of
$E_{\text{reg}}$ with respect to $\gamma$, which is an element of
$T_{\gamma_{\text{id}}}(\Gamma)$ given by
$\partial\gamma\backsimeq\sum_{b_{i}\in{{\mathcal{B}}}_{I}}\langle
h_{1}-\tilde{h}_{2},d\phi(b_{i})\rangle_{2}b_{i}$. This linear combination of
the orthonormal basis elements of $T_{\gamma_{\text{id}}}(\Gamma)$ provides
the incremental update of $\tilde{h}_{2}$ in the orbit $[h_{2}]$.

This leaves two remaining issues: (1) the specification of an orthonormal
basis of $T_{\gamma_{\text{id}}}(\Gamma)$, and (2) an expression for
$\phi_{\ast}$. In our implementation, we use gradients of the spherical
harmonic basis to define the orthonormal basis of
$T_{\gamma_{\text{id}}}(\Gamma)$; see Section 3.2.2 of [21]. The expression of
$\phi_{\ast}$ is also derived in Section 3.2.2 of [21]. With this, the
optimization over the space of diffeomorphisms can be performed using
Algorithm 2.

Input: Two surfaces $\\{f_{1},\ f_{2}\\}\in{\mathcal{F}}$ and small step size
$\epsilon$.
Output: Optimal registration $\gamma^{*}$ and optimally registered surface
$f_{2}^{*}$.

1: Generate basis ${\mathcal{B}}_{I}=\\{b_{i},\ i=1,\dots,N\\}$ using

spherical harmonics.

2: Compute the SRNFs $h_{1}=H(f_{1})$ and $h_{2}=H(f_{2})$.

3: Initialize $\gamma_{0}=\gamma_{id}$, $h_{2}^{0}=h_{2}$ and $j=0$.

4: For each $b_{i},\ i=1,\dots,N$, compute $d\phi(b_{i})$.

5: Compute the registration update
$\partial\gamma=\sum_{b_{i}\in{{\mathcal{B}}}_{I}}\langle
h_{1}-{h}^{j}_{2},d\phi(b_{i})\rangle_{2}b_{i}$.

6: Apply the registration update using
$\gamma_{j+1}=\gamma_{j}\circ(\gamma_{id}+\epsilon\partial\gamma)$.

7: Update $h_{2}^{j+1}=(h_{2}^{0},\gamma_{j+1})$ and $j=j+1$.

8: Iterate steps 4-7 until convergence.

9: Let $\gamma^{*}=\gamma_{j}$ and $f_{2}^{*}=f_{2}\circ\gamma^{*}$.

Algorithm 2 Optimal registration of two surfaces. |
---|---
(a) Without SRNF. | (b) With SRNF.
Figure 13:  The mean shape between the left and right surfaces, computed (a) in the original surface space without the SRNF representation, and (b) in the SRNF space. The computed mean shape is shown in the middle of each subfigure. Observe that in (a), elongated parts such as the arms significantly shrink while in (b) they bend in a natural fashion. |
---|---
(a) Input (before registration). | (b) After registration.
Figure 14:  Illustration of the spatial correspondences, on samples from the CAPE dataset [8], before and after applying the proposed spatial registration algorithm. Correspondences are color-coded, _i.e.,_ points that are in correspondence are rendered with the same color.  |
---|---
(a) Input (before spatial registration). | (b) After spatial registration.
Figure 15:  Illustration of the spatial correspondences, on some samples from
the DFAUST dataset [7], before and after applying the proposed spatial
registration algorithm. Correspondences are color-coded, _i.e.,_ points that
are in correspondence are rendered with the same color.
---
(a) Input (before spatial registration).
(b) After spatial registration.
Figure 16:  Illustration of the spatial correspondences, on samples from the
COMA dataset [46], before and after applying the proposed spatial registration
algorithm. Correspondences are color-coded, _i.e.,_ points that are in
correspondence are rendered with the same color.
---
Figure 17:  Examples of spatial correspondences between objects with missing
parts. Correspondences are color-coded. The top row shows one view of the 3D
models while the bottom row shows another view of the same models. In both
cases, we show the smooth rendering as well as the wireframe rendering of the
surfaces so that the reader can appreciate the quality of the correspondences.
---
Figure 18:  Examples of spatial correspondences between 3D hands with one
finger missing. Correspondences are color-coded, _i.e.,_ points that are in
correspondence are rendered with the same color. We show the smooth rendering
as well as the wireframe rendering so that the reader can appreciate the
quality of the correspondences.

###  A.2 Additional spatial registration results

In this section, we provide more results to show the importance of each
component of the proposed framework and to demonstrate the efficiency of the
proposed spatial registration.

Importance of the SRNF representation. Fig. 13 illustrates, with a practical
example, the importance of the SRNF representation for computing geodesics and
statistics between 3D surfaces that undergo complex articulated and elastic
motion. In this example, we use two human body shapes, perform their spatial
registration using the proposed SRNF framework, and then compute their mean
using the $\mathbb{L}^{2}$ metric in the original space of surfaces (Fig.
13-(a)) and using the $\mathbb{L}^{2}$ metric in the space of SRNFs (Fig.
13-(b)). In the former case, we can see that the parts that undergo large
articulated motion (arms in this example) significantly shrink. This is
predictible since a geodesic under the $\mathbb{L}^{2}$ metric in the original
space of surfaces corresponds to straight lines. In the SRNF space, the arms
naturally bend since the $\mathbb{L}^{2}$ metric in this space is equivalent
to the optimal bending and stretching of surfaces; see Fig. 13-(b).

---
(a) Correspondences using the SRNF framework.
(b) Correspondences using MAPTree [44].
Figure 19:  Examples of spatial correspondences between pairs of quadruped
animals from the ACSM animal dataset [47]. Correspondences are color-coded. We
also show in (a) the wireframe rendering of the surfaces. Correspondences are
color-coded, _i.e.,_ points that are in correspondence are rendered with the
same color. Correspondences are color-coded, _i.e.,_ points that are in
correspondence are rendered with the same color.

Human bodies and faces. Fig. 14 shows three dressed 3D human models, from the
CAPE dataset, before registration (Fig. 14-(a)) and after registration (Fig.
14-(b)) using the proposed SRNF-based approach. In both cases, the
correspondences are color-coded. CAPE dataset is particularly challenging
since it contains dressed 3D human bodies. Despite the significant differences
in the initial parameterizations (see Fig. 14-(a)) and in the shape of the
bodies and clothes, the proposed approach is able to find plausible
correspondences between the surfaces (see Fig. 14-(b)). Figs. 15 and 16 show
more examples using, respectively, the DFAUST human body dataset [7] and the
COMA human faces [46]. In both cases, we show the surface before and after
registration with correspondences color-coded.

Surfaces with missing parts. Figs. 17 and 18 show examples of spatial
registrations between surfaces that have missing parts. Since our framework
uses an elastic metric, which allows bending and stretching of surfaces, it is
able to find one-to-one correspondences even under these challenging cases.
Note that, similar to functional maps, the framework requires closed manifold
surfaces and thus it is not able to handle (noisy) partial scans.

Quadruped animals. We test the proposed SRNF-based spatial registration on the
quadruped animal dataset of [47]. Figs. 19-(a), 20-(a), and 21-(a) show the
correspondence results obtained using our approach (correspondences are color-
coded). We also compare, visually, our results to the registrations obtained
using the state-of-the-art functional map-based methods such as MapTree [44];
see Figs. 20-(b), 19-(b), and 21-(b). As one can see, the mappings obtained
using MapTree are often not correct in many regions of the shapes. More
importantly, the maps obtained using MapTree are not one-to-one since a point
on the source shape can be mapped to multiple points on the target. Our
approach finds one-to-one correspondences and thus is more suitable for
computing geodesics and shape statistics.

---
View 1. View 2.
View 1 with tessellation. View 2 with tessellation.
(a) Correspondences using the SRNF framework.
View 1. View 2.
(b) Correspondences using MAPTree [44].
Figure 20:  Examples of spatial correspondences between quadruped animals from
the ACSM animal dataset [47]. Correspondences are color-coded. We also show in
(a) the tessellation grid of the surfaces. Correspondences are color-coded,
_i.e.,_ points that are in correspondence are rendered with the same color.
---
View 1. View 2.
View 1 with tessellation. View 2 with tessellation.
(a) Correspondences using the SRNF framework.
View 1. View 2.
(b) Correspondences using MAPTree [44].
Figure 21:  Examples of spatial correspondences between quadruped animals from
the ACSM animal dataset [47]. Correspondences are color-coded. We also show in
(a) the tessellation grid of the surfaces. Correspondences are color-coded,
_i.e.,_ points that are in correspondence are rendered with the same color.

###  A.3 Karcher mean of surfaces

Algorithm 3 summarizes the Karcher mean algorithm used to compute the mean 4D
surface of a collections of 4D surfaces; see Section 4 of the main manuscript.
We refer the reader to the main manuscript for the definition of each symbol.

Input: A set of spatially-registered 4D surfaces
$\\{\alpha_{1},\dots,\alpha_{n}\\}\in{\mathcal{M}_{\mathcal{F}}}$ and their
corresponding SRNF trajectories
$\\{\beta_{1},\dots,\beta_{n}\\}\in\mathcal{M}_{h}$.

Output: Karcher mean $\bar{\alpha}\in\mathcal{M}_{\mathcal{F}}$.

1: Compute the TSRVF maps: $q_{i}=Q(\beta_{i}),i=1,\dots,n$.

2: Set $\bar{q}\leftarrow q_{1}$ as an initial estimate of the Karcher mean.

3: Set $\xi_{i},\ i=1,\dots,n$ to the identity $\xi_{id}(t)=t$.

4: For $i=1,\dots,n$,

  * •

Temporally register $q_{i}$ to $\bar{q}$, resulting in
$q^{*}_{i}=q_{i}\odot\xi_{i}^{*}$, by solving Eqn (8) of the main manuscript.

  * •

$q_{i}\leftarrow q^{*}_{i}$ and $\xi_{i}\leftarrow\xi_{i}\circ\xi_{i}^{*}$.

5: Update the Karcher mean $\bar{q}=\frac{1}{n}\sum_{i=1}^{n}q_{i}$.

6: If the change in $\left\|\bar{q}\right\|$ is large then go to Step 4.

7: Find $\bar{\alpha}$ by TSRVF inversion of $\bar{q}$ followed by SRNF
inversion.

8: Return $\bar{\alpha}$, $\xi_{i},q_{i},\i=1,\dots,n$.

Algorithm 3 Karcher mean of 4D surfaces.

##  Appendix B Temporal registration

In this appendix, we provide additional results and additional quantitative
evaluations to further demonstrate the quality and performance of the temporal
registration framework proposed in this paper.

|
---|---
(a) Before registration. | (b) After registration.
Figure 22:  We consider twelve 4D surfaces from the VOCA dataset, perturb each of them using $100$ random temporal diffeomorphisms, and then measure the distance between each original 4D surface and its perturbed versions. We show the boxplots of distances (a) before the temporal registration and (b) after the temporal registration using the proposed framework for each of the twelve 4D surfaces. In each plot, the X axis corresponds to the 4D surfaces and the Y axis to the temporal alignment error. The lower the error is the better is the alignment.  |
---|---
(a) Before registration. | (b) After registration.
Figure 23:  We consider five 4D surfaces generated using FLAME framework,
perturb each of them using $100$ random temporal diffeomorphisms, and then
measure the distance between each original 4D surface and its perturbed
versions. We show the boxplots of distances (a) before the temporal
registration and (b) after the temporal registration using the proposed
framework for each of the five 4D surfaces. In each plot, the X axis
corresponds to the 4D surfaces and the Y axis to the temporal alignment error.
The lower the error is the better is the alignment.

###  B.1 Evaluation of the temporal registration

To quantitatively evaluate the performance of the proposed temporal
registration, we first consider a 4D surface $\alpha$ and parameterize it with
randomly generated temporal diffeomorphims $\xi_{i},i\in\\{1,\cdots,100\\}$,
to obtain new 4D surfaces $\alpha_{i}=\alpha\circ\xi_{i}$. Next, we compute
the distances $d_{i},\ i=1,\dots,100$, between $\alpha$ and $\alpha_{i}$,
using the expression given in Eqn. (7) of the main manuscript, before temporal
registration. Fig. 22-(a) shows the box plots of the resulting distances for
$12$ sequences from the VOCA dataset before the temporal registration.
Ideally, $\forall i,\ d_{i}=0$. However, since the 4D surfaces are not
temporally registered, the distances are large in most of the cases.

In Fig. 23, we perform the same experiment but this time on five 4D surfaces
simulated using the FLAME framework; see Section 5.1 of the main manuscript
for a detailed description of how these 4D surfaces have been generated. Fig.
23 reports the temporal registration errors before and after applying the
proposed framework for each of the five 4D surfaces. As one can see, our
approach is able to properly align the perturbed 4D surfaces to their original
4D surfaces.

Next, we compute the optimal temporal registration, _i.e.,_ for every 4D
surface $\alpha_{i}$, we find the optimal diffeomorphism $\xi^{*}_{i}$ that
aligns $\alpha_{i}$ to $\alpha$. Let
$\alpha^{*}_{i}=\alpha_{i}\circ\xi^{*}_{i}$. Fig. 23-(b) shows boxplots of the
distances between $\alpha^{*}_{i}$ and $\alpha$ for the same five sequences.
Compared to the distances between the original unregistered 4D expressions
shown in Fig. 23-(a), these distances are significantly lower. This shows that
the proposed temporal alignment framework brings the 4D surfaces as close as
possible to each other.

|
---|---
(a) | (b)
Figure 24:  The 100 random temporal diffeomorphisms that have been applied to
the original 4D surfaces to simulate the data (a) in Figs. 22 and 23 of this
supplementary material, and (b) in Fig. 8 of the main manuscript. In each
plot, the X and U axis both correspond to the temporal domain $[0,1]$ since,
in our formulation, a temporal diffeomorphism $\xi$ is defined as the mapping
of the temporal domain $[0,1]$ to itself, _i.e.,_ $\xi:[0,1]\to[0,1]\text{
such that }0<\frac{d\xi}{dt}<\infty,\xi(0)=0\text{ and }\xi(1)=1.$ Figure 25:
Examples of the spatiotemporal registration of two facial expressions. Note
how the spatiotemporally registered target 4D surface in (c) became fully
synchronised with the source 4D surface (a). The full video sequence is
provided in the supplementary material.

Finally, Figs. 24-(a) and (b) show the $100$ random temporal diffeomorphisms
that have been applied to perturb the 4D surfaces used for quantitative
evaluation shown in Fig. 8 of the main manuscript, and in Figs. 22 and 23 in
this Supplementary Material.

###  B.2 Additional temporal registration results

Fig. 25 shows another example of the spatiotemporal registration of two 4D
facial surfaces. In this example, we show the source surface, the target 4D
surface before the spatiotemporal registration, and the target 4D surface
after spatiotemporal registration. We also highlight some key frames to
illustrate the quality of the temporal registration.

##  Appendix C Geodesics between 4D surfaces

This appendix presents additional results on the computation of geodesics
between 4D surfaces. Fig. 26 shows two additional examples of gedoesics
between 4D facial surfaces. Fig. 27, on the other hand, shows the input 4D
surfaces, prior to the spatiotemporal registration, used to generate the mean
4D surface in Fig. 9 in the main manuscript.

---
(a) Example 1.
(b) Example 2.
Figure 26:  Two additional examples of geodesics between 4D surfaces. In each
example, we show the source 4D surface, the target 4D surface after
spatiotemporal registration, and five intermediate 4D surfaces along the
geodesic between the source and the target. The highlight 4D surface
corresponds to the mean. A video illustrating this sequence is included in the
supplementary material. Figure 27:  Input 4D surfaces, prior to the
spatiotemporal registration, used to generate the mean 4D surface of Fig. 9 of
the main manuscript. Each row correspond to one 4D surface. The supplementary
material also includes the full video sequences.

##  Appendix D Statistics between 4D surfaces

Finally, Section D provides additional results on the computation of summary
statistics on collections of 4D surfaces. Figs. 28 and 29 show statistics
computed on a collection of six 4D facial expressions from the VOCA dataset.
In Fig. 28, we show the input 4D surfaces before and after the spatiotemporal
registration. In both cases, we highlight the computed mean 4D surface. Fig.
29 shows the leading three modes of variation; each mode of variation is
discretized as a path from $-1.5$ to $+1.5$ standard deviations.

---
(a) Input unregistered 4D surfaces. The last row is the mean obtained without
temporal registration.
(b) Spatio-temporally co-registered 4D surfaces. The last row is the mean
obtained after co-registration
The mean is shown in the last column.
Figure 28:  Input 4D faces and mean 4D surface computed before (a) and after
(b) co-registration. Each row corresponds to one 4D surfaces. The
supplementary material includes the full video sequences.
---
(a) First mode of variation.
(b) Second mode of variation.
(c) Third mode of variation.
Figure 29:  Three principal modes of variation (the mean 4D surface is
highlighted in the middle). Each row corresponds to one 4D surface. The
supplementary material includes the full video sequences.

## References

  * [1] P. Werner, D. Lopez-Martinez, S. Walter, A. Al-Hamadi, S. Gruss, and R. Picard, “Automatic recognition methods supporting pain assessment: A survey,” _IEEE Transa. Affective Computing_ , 2019.
  * [2] B. Egger, W. A. Smith, A. Tewari, S. Wuhrer, M. Zollhoefer, T. Beeler, F. Bernard, T. Bolkart, A. Kortylewski, S. Romdhani _et al._ , “3D Morphable Face Models: Past, Present, and Future,” _ACM TOG_ , vol. 39, no. 5, pp. 1–38, 2020.
  * [3] B. Allen, B. Curless, and Z. Popović, “The space of human body shapes: Reconstruction and parameterization from range scans,” _ACM Transactions on Graphics_ , vol. 22, no. 3, pp. 587–594, Jul. 2003.
  * [4] H. Laga, Q. Xie, I. H. Jermyn, and A. Srivastava, “Numerical inversion of srnf maps for elastic shape analysis of genus-zero surfaces,” _IEEE PAMI_ , vol. 39, no. 12, pp. 2451–2464, 2017.
  * [5] I. Jermyn, S. Kurtek, E. Klassen, and A. Srivastava, “Elastic shape matching of parameterized surfaces using square root normal fields,” _ECCV_ , vol. 5, no. 14, pp. 805–817, 2012.
  * [6] D. Cudeiro, T. Bolkart, C. Laidlaw, A. Ranjan, and M. Black, “Capture, learning, and synthesis of 3D speaking styles,” _IEEE CVPR_ , pp. 10 101–10 111, 2019. [Online]. Available: http://voca.is.tue.mpg.de/
  * [7] F. Bogo, J. Romero, G. Pons-Moll, and M. J. Black, “Dynamic FAUST: Registering human bodies in motion,” in _IEEE CVPR_ , 2017.
  * [8] Q. Ma, J. Yang, A. Ranjan, S. Pujades, G. Pons-Moll, S. Tang, and M. J. Black, “Learning to dress 3d people in generative clothing,” in _IEEE CVPR_ , 2020, pp. 6469–6478.
  * [9] T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham, “Active shape models-their training and application,” _Computer vision and image understanding_ , vol. 61, no. 1, pp. 38–59, 1995.
  * [10] V. Blanz and T. Vetter, “A morphable model for the synthesis of 3d faces,” in _ACM Siggraph_ , 1999, pp. 187–194.
  * [11] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, and J. Davis, “Scape: shape completion and animation of people,” in _ACM SIGGRAPH 2005 Papers_ , 2005, pp. 408–416.
  * [12] N. Hasler, C. Stoll, M. Sunkel, B. Rosenhahn, and H.-P. Seidel, “A statistical model of human pose and body shape,” in _CGF_ , vol. 28, no. 2, 2009, pp. 337–346.
  * [13] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, “Smpl: A skinned multi-person linear model,” _ACM TOG_ , vol. 34, no. 6, pp. 1–16, 2015.
  * [14] S. Zuffi, A. Kanazawa, T. Berger-Wolf, and M. J. Black, “Three-d safari: Learning to estimate zebra pose, shape, and texture from images,” in _IEEE CVPR_ , 2019, pp. 5359–5368.
  * [15] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. Osman, D. Tzionas, and M. J. Black, “Expressive body capture: 3d hands, face, and body from a single image,” in _IEEE CVPR_ , 2019, pp. 10 975–10 985.
  * [16] N. Hesse, S. Pujades, J. Romero, M. J. Black, C. Bodensteiner, M. Arens, U. G. Hofmann, U. Tacke, M. Hadders-Algra _et al._ , “Learning an infant body model from rgb-d data for accurate full body motion analysis,” in _MICCAI_ , 2018, pp. 792–800.
  * [17] S. Kurtek, A. Srivastava, E. Klassen, and H. Laga, “Landmark-guided elastic shape analysis of spherically-parameterized surfaces,” _CGF_ , vol. 32, no. 2pt4, pp. 429–438, 2013.
  * [18] M. Kilian, N. J. Mitra, and H. Pottmann, “Geometric modeling in shape space,” in _ACM SIGGRAPH_ , 2007.
  * [19] Q. Xie, S. Kurtek, H. Le, and A. Srivastava, “Parallel transport of deformations in shape space of elastic surfaces,” in _IEEE ICCV_ , December 2013.
  * [20] Q. Xie, I. Jermyn, S. Kurtek, and A. Srivastava, “Numerical inversion of srnfs for efficient elastic shape analysis of star-shaped objects,” in _ECCV_ , 2014, pp. 485–499.
  * [21] I. H. Jermyn, S. Kurtek, H. Laga, and A. Srivastava, “Elastic shape analysis of three-dimensional objects,” _Synthesis Lectures on Computer Vision_ , vol. 12, no. 1, pp. 1–185, 2017.
  * [22] H. Laga, “A survey on nonrigid 3D shape analysis,” in _Academic Press Library in Signal Processing, Volume 6_. Elsevier, 2018, pp. 261–304.
  * [23] H. Laga, Y. Guo, H. Tabia, R. B. Fisher, and M. Bennamoun, _3D Shape analysis: fundamentals, theory, and applications_. John Wiley & Sons, 2018.
  * [24] A. M. Bronstein, M. M. Bronstein, and R. Kimmel, “Generalized multidimensional scaling: a framework for isometry-invariant partial surface matching,” _Proceedings of the National Academy of Sciences_ , vol. 103, no. 5, pp. 1168–1172, 2006.
  * [25] R. Litman and A. M. Bronstein, “Learning spectral descriptors for deformable shape correspondence,” _IEEE PAMI_ , vol. 36, no. 1, pp. 171–180, 2013.
  * [26] M. Ovsjanikov, M. Ben-Chen, J. Solomon, A. Butscher, and L. Guibas, “Functional maps: a flexible representation of maps between shapes,” _ACM TOG_ , vol. 31, no. 4, pp. 1–11, 2012.
  * [27] M. Ovsjanikov, E. Corman, M. Bronstein, E. Rodolà, M. Ben-Chen, L. Guibas, F. Chazal, and A. Bronstein, “Computing and processing correspondences with functional maps,” in _SIGGRAPH ASIA 2016 Courses_ , 2016, pp. 1–60.
  * [28] M. Wand, P. Jenke, Q. Huang, M. Bokeloh, L. Guibas, and A. Schilling, “Reconstruction of deforming geometry from time-varying point clouds,” in _SGP_ , 2007, pp. 49–58.
  * [29] T. Beeler, F. Hahn, D. Bradley, B. Bickel, P. Beardsley, C. Gotsman, R. W. Sumner, and M. Gross, “High-quality passive facial performance capture using anchor frames,” in _ACM SIGGRAPH_ , 2011, pp. 1–10.
  * [30] A. Tevs, A. Berner, M. Wand, I. Ihrke, M. Bokeloh, J. Kerber, and H.-P. Seidel, “Animation cartography?intrinsic reconstruction of shape and motion,” _ACM TOG_ , vol. 31, no. 2, pp. 1–15, 2012.
  * [31] T. Li, T. Bolkart, M. J. Black, H. Li, and J. Romero, “Learning a model of facial shape and expression from 4d scans.” _ACM Trans. Graph._ , vol. 36, no. 6, pp. 194–1, 2017.
  * [32] R. Anirudh, P. Turaga, J. Su, and A. Srivastava, “Elastic functional coding of human actions: From vector-fields to latent variables,” in _IEEE CVPR_ , 2015, pp. 3147–3155.
  * [33] I. Akhter, T. Simon, S. Khan, I. Matthews, and Y. Sheikh, “Bilinear spatiotemporal basis models,” _ACM Transactions on Graphics (TOG)_ , vol. 31, no. 2, pp. 1–12, 2012.
  * [34] B. B. Amor, J. Su, and A. Srivastava, “Action recognition using rate-invariant analysis of skeletal shape trajectories,” _IEEE PAMI_ , vol. 38, no. 1, pp. 1–13, 2015.
  * [35] I. Dryden and K. Mardia, _Statistical Shape Analysis_. John Wiley & Son, 1998.
  * [36] A. Srivastava, E. Klassen, S. Joshi, and I. Jermyn, “Shape analysis of elastic curves in euclidean spaces,” _IEEE PAMI_ , no. 99, pp. 1–1, 2011.
  * [37] M. F. Beg, M. I. Miller, A. Trouvé, and L. Younes, “Computing large deformation metric mappings via geodesic flows of diffeomorphisms,” _IJCV_ , vol. 61, no. 2, pp. 139–157, 2005.
  * [38] V. Debavelaere, S. Durrleman, S. Allassonnière, and A. D. N. Initiative, “Learning the Clustering of Longitudinal Shape Data Sets into a Mixture of Independent or Branching Trajectories,” _IJCV_ , pp. 1–16, 2020.
  * [39] A. Bône, O. Colliot, and S. Durrleman, “Learning the Spatiotemporal Variability in Longitudinal Shape Data Sets,” _IJCV_ , vol. 128, no. 12, pp. 2873–2896, 2020.
  * [40] E. Praun and H. Hoppe, “Spherical parametrization and remeshing,” vol. 22, 2003, pp. 340–349.
  * [41] S. Kurtek, E. Klassen, Z. Ding, S. Jacobson, J. Jacobson, M. Avison, and A. Srivastava, “Parameterization-invariant shape comparisons of anatomical surfaces,” _Medical Imaging, IEEE Transactions on_ , vol. 30, no. 3, pp. 849–858, 2011.
  * [42] J. Su, S. Kurtek, E. Klassen, A. Srivastava _et al._ , “Statistical analysis of trajectories on Riemannian manifolds: bird migration, hurricane tracking and video surveillance,” _The Annals of Applied Statistics_ , vol. 8, no. 1, pp. 530–552, 2014.
  * [43] D. T. Robinson, _Functional data analysis and partial shape matching in the square root velocity framework_. The Florida State University, 2012.
  * [44] J. Ren, S. Melzi, M. Ovsjanikov, and P. Wonka, “Maptree: recovering multiple solutions in the space of maps,” _ACM Transactions on Graphics (TOG)_ , vol. 39, no. 6, pp. 1–17, 2020.
  * [45] G. Pai, J. Ren, S. Melzi, P. Wonka, and M. Ovsjanikov, “Fast sinkhorn filters: Using matrix scaling for non-rigid shape correspondence with functional maps,” in _CVPR_ , 2021.
  * [46] A. Ranjan, T. Bolkart, S. Sanyal, and M. J. Black, “Generating 3D faces using convolutional mesh autoencoders,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 704–720.
  * [47] N. Kulkarni, A. Gupta, D. F. Fouhey, and S. Tulsiani, “Articulation-aware canonical surface mapping,” in _IEEE CVPR_ , 2020, pp. 452–461.
  * [48] M. Bauer, N. Charon, P. Harms, and H.-W. Hsieh, “A numerical framework for elastic surface matching, comparison, and interpolation,” _International Journal of Computer Vision_ , pp. 1–20, 2021.
  * [49] G. Wang, H. Laga, N. Xie, J. Jia, and H. Tabia, “The shape space of 3d botanical tree models,” _ACM TOG_ , vol. 37, no. 1, pp. 1–18, 2018.
  * [50] G. Wang, H. Laga, J. Jia, S. J. Miklavcic, and A. Srivastava, “Statistical analysis and modeling of the geometry and topology of plant roots,” _J. of Theoretical Biology_ , vol. 486, p. 110108, 2020.
  * [51] X. Han, H. Laga, and M. Bennamoun, “Image-based 3d object reconstruction: State-of-the-art and trends in the deep learning era,” _IEEE PAMI_ , 2019\.
  * [52] H. Laga, L. V. Jospin, F. Boussaid, and M. Bennamoun, “A survey on deep learning techniques for stereo-based depth estimation,” _IEEE PAMI_ , 2020\.