# Augmenting Implicit Neural Shape Representations
with Explicit Deformation Fields

Matan Atzmon Work partially done during an internship at Facebook AI Research.
Weizmann Institute of Science  David Novotny  Facebook AI Research  Andrea
Vedaldi  Facebook AI Research  Yaron Lipman  Weizmann Institute of Science
Facebook AI Research

###### Abstract

Implicit neural representation is a recent approach to learn shape collections
as zero level-sets of neural networks, where each shape is represented by a
latent code. So far, the focus has been shape reconstruction, while shape
generalization was mostly left to generic encoder-decoder or auto-decoder
regularization. In this paper we advocate deformation-aware regularization for
implicit neural representations, aiming at producing plausible deformations as
latent code changes. The challenge is that implicit representations do not
capture correspondences between different shapes, which makes it difficult to
represent and regularize their deformations. Thus, we propose to pair the
implicit representation of the shapes with an explicit, piecewise linear
deformation field, learned as an auxiliary function. We demonstrate that, by
regularizing these deformation fields, we can encourage the implicit neural
representation to induce natural deformations in the learned shape space, such
as as-rigid-as-possible deformations.

Figure 1: Augmenting implicit neural representations with consistent
deformation fields allows incorporating deformation priors, _e.g_., piecewise
rigid, to shape space interpolations (top row); bottom row depicts state of
the art implcit neural interpolation baseline; far left and right depict input
raw scans.

##  1 Introduction

The focal point of this work are implicit surfaces, defined as zero level-sets
of scalar functions $f:\mathbb{R}^{3}\rightarrow\mathbb{R}$,

${\mathcal{S}}=\left\\{{\bm{x}}\in\mathbb{R}^{3}\ |\ f({\bm{x}})=0\right\\}.$

The main benefit of an implicit shape representation is its ability to capture
detailed continuous surfaces with arbitrary topology without requiring
discretization, complex data structures, or explicit sampling.

Implicit surfaces recently found good pairing with neural networks.
Representing an implicit function such as $f$ with a neural network allows to
leverage its expressive power to model accurately complex 3D scenes and
objects [10, 3, 26, 35]. Neural implicit surfaces can also represent
_collections_ of 3D objects with different but related shapes. This is
typically operationalized by conditioning the implicit function $f$ with a
latent code ${\bm{z}}\in\mathbb{R}^{D}$:

${\mathcal{S}}({\bm{z}})=\left\\{{\bm{x}}\in\mathbb{R}^{3}\ |\
f({\bm{x}},{\bm{z}})=0\right\\}.$

By varying ${\bm{z}}$, we can adjust the function’s zero level-set to match
different members of the shape set. Indeed, recent works [28, 25, 31] have
demonstrated that, in this manner, it is possible to encode collections of
complex 3D shapes of humans or human-made objects by low-dimensional latent
code vectors.

Previous works have focused on the problem of obtaining high-fidelity
reconstructions of a given set of 3D shapes form of noisy or incomplete
measurements such as point clouds or triangle soups. However, they have
usually paid less attention to the shapes ${\mathcal{S}}({\bm{z}})$ generated
by intermediate latent vectors ${\bm{z}}$. For example, given that
${\mathcal{S}}({\bm{z}}_{1})$ and ${\mathcal{S}}({\bm{z}}_{2})$ reconstruct
two shapes from the input dataset, do
${\mathcal{S}}((1-t){\bm{z}}_{1}+t{\bm{z}}_{2})$, $t\in(0,1)$, form plausible
intermediate shapes as well? In this paper we answer this question, with an
aim to not just reconstructing a given set of shapes from sparse observations,
but also to generalize the given examples to obtain a full shape space.

Previous works on learning shape spaces have usually relied on generic latent-
space regularizers such as VAE [25, 3] or Auto-Decoders (AD) [31] that mainly
enforce smoothness of latent code transitions. However, in many cases
smoothness alone is not sufficient to learn plausible shapes. For example, in
Figure 1 the bottom row shows implicit surfaces generated by interpolating the
latent code obtained from state of the art implicit neural shape
representations, demonstrating typical defects such as parts popping in and
out of existence rather than deforming in a plausible manner.

Our main contribution is a new shape space regularization framework that
encourages implicit neural shape representations to generate plausible
interpolations of the training shapes. Intuitively, we would like different
shapes to be related by plausible deformations, such as the one arising from
articulation of the underlying physical object. However, differently from
explicit shape representations such as meshes, implicit shapes do not provide
point correspondences between deformations, which makes it difficult to apply
standard geometric regularizers.

In order to address this challenge, we start from a careful analysis of the
way neural level sets deform as the latent code changes. Inspired by classic
level-set tracking methods [39, 41], we parameterize the collection of (non-
unique) explicit vector fields, called _deformation fields_ , that are
consistent with the level-set deformation for a particular latent code change.
Then, among these deformation fields, we seek the one that minimizes a
geometric loss, consequently enforcing a geometric prior on the implicit
neural representation $f$. We use in particular the as-rigid-as-possible
energy, also known as Killing energy [6, 37, 36, 14, 13], due to its ability
to encourage natural elastic or piece-wise rigid shape deformations.

Although we opt for a specific geometric prior in this paper, an important
advantage of our framework is its generality: it allows to incorporate any
desirable deformation prior, which is expressible via explicit deformation
fields, on the implicit neural shape space representation.

Empirically, we show that our framework, applied on the _raw_ D-Faust dataset
[7], can learn a plausible shape space, considerably improving upon existing
baselines (see _e.g_. Figure 1 top vs bottom row). The power of our method is
that, given unstructured and sparse data such as a triangle soup, it
simultaneously achieves high-quality surface reconstructions, similar to
previous work utilizing implicit shape representations, while also learning a
space of plausible shapes generalizing the training samples. We provide both
quantitative and qualitative results in the experimental section.

##  2 Related Work

###  2.1 Shape representations

Since our work considers learning of a shape space, below we revise the most
relevant 3D shape representations.

##### Implicit representations.

Implicit shape representations define a 3D shape as a level-set of a scalar
function on $\mathbb{R}^{3}$ (equation 1). Several works [43, 48, 17, 11] have
explored representing such functions with voxel grids, which are implicit
functions evaluated at 3D grids. Implicit functions on a continuous domain
have been explored later [31, 49, 3, 2, 9, 25]. Park et al. [31] trained deep
networks to label each 3D point with its signed distance from the nearest
surface point on the boundary of a shape producing a signed distance field
(SDF). Disn [49] improved the architecture of [31]. Preceding the deep
learning era, SDFs were successfully applied to represent scene geometries in
[27]. Atzmon et al. [3] fitted signed distance fields to raw data via a sign
agnostic similarity loss, and [2] proposed to improve SDF learning via a
differentiable parametrization of samples from the neural level-set. Deep
occupancy fields, a specific kind of neural SDFs which reduce the distance
function only to its sign, have been proposed in [9, 25].

Several works model shapes as compositions of well-defined 3D primitives.
Genova et al. [16, 15] learn structured deep representations whose main
components are Gaussian-shaped occupancy functions. Compositions of more
explicitly defined mesh primitives were introduced in [23]. [46, 29] who split
the 3D domain into Voronoi primitives and express their occupancies to
delineate the surface.

The aforementioned approaches mostly improve the expressiveness of the
representations to better model shape details. As such, our contribution is
complimentary, as we focus on regularizing an arbitrary neural level-set
predictor in order to improve generalization to test data.

##### Explicit surfaces.

Instead of defining a surface implicitly as a function’s level-set, others
explored more explicit representations, such as meshes. Notable deep mesh
predictors were introduced in [19, 44]. [20, 47] proposed a more general
representation that maps points from a fixed topological space (a 2D square)
to 3D euclidean space. Others [42, 34, 24] represent shapes as a union over
multi-view back-projections of the visible part of its surface.

###  2.2 Shape interpolation

Shape interpolation studies the problem of non-rigidly deforming a source 3D
shape such that it matches a target template. In principle, there are
infinitely many possible displacement trajectories that “flow” one shape to
the other, which calls for an efficient regularizer that selects the most
natural solution.

##### ARAP surface regularization.

Similar to others, our work exploits the seminal as-rigid-as-possible (ARAP)
paradigm [38] that constrains deformations to be as isometric as possible. In
the context of vector fields that displace implicit surfaces, ARAP can be
operationalized with the Killing energy [6]. A flow field with minimal Killing
energy is denoted as-Killing-as-possible vector field (KVF) [6]. Solomon et
al. [37] applied such KVFs to regularize 2D image warps.

KVFs were further applied to implicit surface tracking, which identifies
trajectories of points attached to a time-evolving surface. The as-killing-as-
possible constraint aided a surface point tracker in [41]. Previously, [39]
proposed to track points with constant normal. Similar to [39, 41], we tie
level-set points to move consistently on the surface. We differ from these
works by using KVFs to learn the implicit surfaces’ _shape and motion_ ,
rather than just tracking points on a given sequence. This introduces several
new challenges such as the requirement of an explicit parameterization of
consistent vectors fields and efficient (closed form) solution for the KVF at
sampled implicit surfaces.

##### Correspondence flows.

Slavcheva et al. [36] find a KVF that matches implicit surfaces of a pair of
3D scans. Eisenberger et al. [14] interpolate shapes by constructing a volume-
preserving flow and minimizing a reconstruction and ARAP energies. [13]
improves [14] by introducing additional momentum conservation constraints.
Among learning-based methods, Limp [12] trains a mesh deformation network
given ground truth 3D correspondences. [21, 45] differ from Limp by minimizing
the Chamfer distance which does not require correspondence annotations. Our
method is also unsupervised and takes as input raw 3D scans. Furthermore, we
are the first to jointly consider the shape interpolation in the context of
implicit neural shape space learning.

##  3 Method

Given a collection of input geometries,
${\mathcal{X}}^{(i)}\subset\mathbb{R}^{3}$,
$i\in[m]:=\left\\{1,2,\ldots,m\right\\}$ (_e.g_. manifold meshes, raw point-
clouds, or triangle soups), our goal is to learn a _shape space_. In our case,
this means a neural network
$f:\mathbb{R}^{3}\times\mathbb{R}^{D}\rightarrow\mathbb{R}$, where each latent
vector ${\bm{z}}\in\mathbb{R}^{D}$ represents a shape given by the implicitly-
defined surface

${\mathcal{S}}({\bm{z}})=\left\\{{\bm{x}}\in\mathbb{R}^{3}\ |\
f_{\theta}({\bm{x}},{\bm{z}})=0\right\\},$ (1)

where $\theta\in\mathbb{R}^{p}$ are the learnable network parameters; in the
following we will drop the $\theta$ subscript of $f$ for brevity. Two
important properties of the shape space are: (i) All input shapes are
represented, _i.e_., for all shapes
${\mathcal{S}}({\bm{z}}^{(i)})\approx{\mathcal{X}}^{(i)}$ for some
${\bm{z}}^{(i)}\in\mathbb{R}^{d}$; and (ii) other latent vectors
${\bm{z}}\in\mathbb{R}^{D}$ correspond to _plausible_ shape deformations. For
example, a shape ${\mathcal{S}}(t{\bm{z}}^{(i)}+(1-t){\bm{z}}^{(j)})$ for
$t\in(0,1)$ would correspond to a natural deformation between
${\mathcal{S}}({\bm{z}}^{(i)})$ and ${\mathcal{S}}({\bm{z}}^{(j)})$.

The first requirement of the shape spaces (i) is enforced as in previous works
(detailed later). The main focus of this work is (ii), _i.e_., regularizing
$f$ so to enforce plausible shape deformations as latent vectors change; this
is done using consistent deformation fields, described next.

###  3.1 Consistent deformation fields

Consider a smooth change in latent codes ${\bm{z}}(t)$,
$t\in(-{\epsilon},{\epsilon})$, where
${\bm{z}}(0)={\bm{z}}_{0}\in\mathbb{R}^{D}$ is the position at time $t=0$, and
$\dot{{\bm{z}}}(0)={\bm{\eta}}\in\mathbb{R}^{D}$ is the speed at $t=0$.
Integrated, this process generates a curve in latent space describing a
deformation of the shapes, ${\mathcal{S}}({\bm{z}}(t))$. We would like to
incorporate a prior on this deformation. For example, in many applications the
shape deformations are expected to be locally approximately rigid (_e.g_.,
because the deformation is driven by an articulated skeleton).

The challenge of imposing such a rigidity constraint is that the shapes are
given to us implicitly as zero level-sets of a neural network,
$f({\bm{x}},{\bm{z}}(t))$. Differently from the case of meshes, where the
trajectory of individual vertices is known, with implicit surfaces the motion
of 3D points is only known up to a shift along the surface. We thus propose to
supplement the implicit representation with explicit consistent vector fields
${\bm{v}}$ of point displacements that provide all possible explanations of
the deformation of the surface ${\mathcal{S}}({\bm{z}}(t))$, and use the
latter to define deformation priors. Consistent vector fields have been
previously considered for tracking known/input sequences of implicit surfaces
[39, 41]. Here, we use them to facilitate the learning of a space of
deformable implicit surfaces.

We call a vector field
${\bm{v}}={\bm{v}}_{{\bm{z}}_{0},{\bm{\eta}}}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{3}$
_consistent_ with the latent motion ${\bm{z}}(t)$ if, for all
${\bm{x}}\in{\mathcal{S}}({\bm{z}}_{0})$, it satisfies the constraint

$\frac{d}{dt}\Big{|}_{t=0}f({\bm{x}}+t{\bm{v}}({\bm{x}}),{\bm{z}}(t))=0.$ (2)

Figure 2:  Consistent deformation field.

Intuitively, this means that, given a point ${\bm{x}}$ that belongs to the
implicit surface at time $0$, _i.e_.,
${\bm{x}}\in{\mathcal{S}}({\bm{z}}_{0})$, then moving ${\bm{x}}$ at speed
${\bm{v}}({\bm{x}})$ will keep the point on the surface
${\mathcal{S}}({\bm{z}}(t))$ for small (infinitesimal) times $t$. The chain
rule allows writing eq. 2 more explicitly as

$\nabla_{\bm{x}}f~{}{\bm{v}}+\frac{\partial
f}{\partial{\bm{z}}}{\bm{\eta}}=0,$ (3)

where $f$ is evaluated at $({\bm{x}},{\bm{z}}_{0})$, and
$\nabla_{\bm{x}}f\in\mathbb{R}^{1\times 3}$ denotes the gradient of $f$.
Equation 3 is an under-determined linear system in ${\bm{v}}\in\mathbb{R}^{3}$
and its general solution can be written as a sum of a particular solution and
the kernel of $\nabla_{\bm{x}}f({\bm{x}},{\bm{z}}_{0})$. Specifically, we
write ${\bm{v}}={\bm{w}}+{\bm{w}}^{\perp}$ where
${\bm{w}}={\bm{w}}_{{\bm{z}}_{0},{\bm{\eta}}}$ is the particular solution to
eq. 3 given by

${\bm{w}}=-\frac{\nabla_{\bm{x}}f^{T}}{\left\|\nabla_{\bm{x}}f\right\|^{2}}\frac{\partial
f}{\partial{\bm{z}}}{\bm{\eta}},$ (4)

and ${\bm{w}}^{\perp}$ is any vector perpendicular to
$\nabla_{\bm{x}}f({\bm{x}},{\bm{z}}_{0})$. We let
${\bm{w}}^{\perp}={\bm{P}}{\bm{u}}$, where
${\bm{u}}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{3}$ is an arbitrary vector
field, and ${\bm{P}}$ is the projection matrix on $\nabla_{\bm{x}}f^{\perp}$
(_i.e_., the tangent space to ${\mathcal{S}}({\bm{z}}_{0})$ at ${\bm{x}}$)
defined by

${\bm{P}}={\bm{I}}-\frac{\nabla_{\bm{x}}f^{T}\nabla_{\bm{x}}f}{\left\|\nabla_{\bm{x}}f\right\|^{2}},$
(5)

where ${\bm{I}}\in\mathbb{R}^{3\times 3}$ is the identity matrix. In this
manner, we can write the space of consistent vector fields, i.e. the fields

${\bm{v}}={\bm{v}}_{{\bm{z}}_{0},{\bm{\eta}},{\bm{u}}}:{\mathcal{S}}({\bm{z}}_{0})\rightarrow\mathbb{R}^{3}$

that satisfy eq. 3, as

${\bm{v}}={\bm{w}}+{\bm{P}}{\bm{u}}.$ (6)

Intuitively, eq. 6 represents the space of possible motions of the surface
points that are consistent with the implicit shape deformation, where the
ambiguity here comes from the fact that a point ${\bm{x}}$ can also move
arbitrarily in the tangent directions to the surface
${\mathcal{S}}({\bm{z}}_{0})$ without compromising consistency.

###  3.2 Deformation priors

Now that consistent vector fields ${\bm{v}}$ are introduced, we are ready to
impose deformation priors on the shape space. The deformation priors will take
the form

$\text{loss}_{\text{d}}(\theta)=\min_{{\bm{u}}}\mathbb{E}_{{\bm{z}},{\bm{\eta}},{\bm{x}}}\,\rho({\bm{x}};{\bm{v}}_{{\bm{z}},{\bm{\eta}},{\bm{u}}}),$
(7)

where the latent code ${\bm{z}}$ and variation ${\bm{\eta}}$ are sampled
according to some distribution
${\mathcal{D}}={\mathcal{D}}(\mathbb{R}^{D}\times\mathbb{R}^{D})$, the 3D
point ${\bm{x}}$ is sampled from some distribution over the corresponding
surface ${\mathcal{S}}({\bm{z}})$, and $\rho({\bm{x}};{\bm{v}})$ is a measure
of deformation induced by the field ${\bm{v}}$ at location ${\bm{x}}$. As
commonly done, we approximate the expectation in eq. 7 with the empirical
expectation

$\mathrm{loss}_{\text{d}}(\theta)=\min_{\bm{u}}\frac{1}{n}\sum_{i=1}^{n}\rho({\bm{x}}_{i};{\bm{v}}_{{\bm{z}}_{i},{\bm{\eta}}_{i},{\bm{u}}}),$
(8)

where we consider $n$ samples,
$({\bm{z}}_{i},{\bm{\eta}}_{i})\sim{\mathcal{D}}$ and
${\bm{x}}_{i}\in{\mathcal{S}}({\bm{z}}_{i})$, $i\in[n]$. Since all choices of
${\bm{u}}$ correspond to consistent deformation fields, taking the minimum
w.r.t. ${\bm{u}}$ is important to ensure we are considering the deformation
field with lowest deformation error explaining the current shape space
deformation ${\bm{z}}(t)$. In the following example, we demonstrate why taking
an arbitrary consistent vector field instead of the optimal one is
insufficient.

##### Example.

We provide a simple example of a rigid movement of an implicit shape with a
non rigid consistent deformation field. Consider the implicit function

$f({\bm{x}},t)=\left\|{\bm{x}}-t{\bm{\eta}}\right\|^{2}-1,$

representing a unit sphere translated with constant velocity ${\bm{\eta}}$.
The consistency eq. 3 evaluated at $t=0$ in this case is

$\left\langle{\bm{x}},{\bm{v}}-{\bm{\eta}}\right\rangle=0$

for all ${\bm{x}}\in{\mathcal{S}}(0)$, _i.e_., $\left\|{\bm{x}}\right\|=1$.
Therefore
${\bm{w}}=\frac{\left\langle{\bm{x}},{\bm{\eta}}\right\rangle}{\left\|{\bm{x}}\right\|^{2}}{\bm{x}}$

Figure 3:  Non-rigid consistent field ${\bm{v}}$.

is a particular solution to the consistency equation, see fig. 3 (in green).
Now consider the two points on the sphere
${\bm{x}}_{1}=\frac{{\bm{\eta}}}{\left\|{\bm{\eta}}\right\|}$ and
${\bm{x}}_{2}=\frac{{\bm{\eta}}^{\perp}}{\left\|{\bm{\eta}}\right\|}$, where
${\bm{\eta}}^{\perp}$ is an orthogonal vector to ${\bm{\eta}}$, so that
${\bm{w}}({\bm{x}}_{1})={\bm{\eta}}$ while ${\bm{w}}({\bm{x}}_{2})=0$. Thus,
while the surface is moving rigidly with uniform velocity ${\bm{\eta}}$, we
have found an “alternative” velocity field ${\bm{w}}$ that is compatible with
the change in the implicit function, but that causes the distance between the
surface points ${\bm{x}}_{1}$ a ${\bm{x}}_{2}$ to change. We can recover the
“true” velocity field by setting the auxiliary vector
${\bm{u}}({\bm{x}}_{i})={\bm{\eta}}$, which results in
${\bm{v}}({\bm{x}}_{i})={\bm{w}}({\bm{x}}_{i})+{\bm{P}}{\bm{u}}({\bm{x}}_{i})={\bm{\eta}}$
for $i=1,2$.

##### Killing energy.

The Killing energy [37, 41, 36] is applied to vector fields and measures their
deviation from vector fields induced from rigid motions, making it an obvious
choice of regularizer in our case. The Killing energy of a vector field
${\bm{v}}$ is defined as

$\rho({\bm{x}};{\bm{v}})=\left\|\nabla_{\bm{x}}{\bm{v}}({\bm{x}})+\nabla_{\bm{x}}{\bm{v}}({\bm{x}})^{T}\right\|^{2}_{F}.$
(9)

##### Rigid deformation prior.

Next, we look at the problem of predicting the auxiliary vector field
${\bm{u}}({\bm{x}})$. For this, we may use a sophisticated regressor such as a
second neural network, but this is prone to overfitting. From the example
above, ${\bm{u}}$ should approximate the velocity field of the object, which
is locally rigid. If the motion was globally rigid (_i.e_.,
${\bm{x}}(t)=R_{t}{\bm{x}}(0)+T_{t}$ for time-varying $(R_{t},T_{t})\in
SE(3)$), then it could be explained by an affine choice of velocity field:

${\bm{u}}({\bm{x}})={\bm{A}}{\bm{x}}+{\bm{b}},$ (10)

where $({\bm{A}},{\bm{b}})\in\mathbb{R}^{3\times 4}$ (this can be seen by
differentiating ${\bm{x}}(t)$ with respect to time). Now plugging eq. 10 in
eq. 6 and taking the derivative w.r.t. ${\bm{x}}$ gives:

$\nabla_{\bm{x}}{\bm{v}}=\nabla_{\bm{x}}{\bm{w}}+(D_{\bm{x}}{\bm{P}})({\bm{A}}{\bm{x}}+{\bm{b}})+{\bm{P}}{\bm{A}},$

where, for a vector ${\bm{q}}=(q_{1},q_{2},q_{3})^{T}$, we denote
$\left[(D_{\bm{x}}{\bm{P}}){\bm{q}}\right]_{i}=\sum_{j=1}^{3}\nabla_{\bm{x}}P_{ij}({\bm{x}})q_{j}$.
Using this in eqs. 9 and 8 leads to a least-squares (LS) problem in
${\bm{A}},{\bm{b}}$. Minimization w.r.t. ${\bm{A}},{\bm{b}}$ is done by
solving the normal equations. Let ${\bm{A}}_{\star},{\bm{b}}_{\star}$ denote
the minimizer of this LS problem, so that
${\bm{u}}_{\star}({\bm{x}})={\bm{A}}_{\star}{\bm{x}}+{\bm{b}}_{\star}$.
Plugging ${\bm{u}}_{\star}$ in eq. 8 yields

$\mathrm{loss}_{\text{d}}(\theta)=\frac{1}{n}\sum_{i=1}^{n}\rho({\bm{x}}_{i};{\bm{v}}_{{\bm{z}}_{i},{\bm{\eta}}_{i},{\bm{u}}_{\star}}).$
(11)

Clearly, the two losses in equations eq. 11 and eq. 8 share the same value at
$\theta$. Furthermore, Danskin’s Theorem (see supplamentary) asserts that they
also have the same gradient at $\theta$.

##### Multiple rigid deformation priors.

A single affine ${\bm{u}}$ can explain a global rigid motion of the implicit
shape. Combining a collection of such affine fields can be used to introduce a
piecewise rigid motion prior. In order to do so, we introduce probability
vector, also modeled with a neural network (we will continue to denote the
collection of all learnable parameters by $\theta$),
${\bm{p}}=(p_{1},\ldots,p_{k}):\mathbb{R}^{3}\rightarrow[0,1]^{k}$, where $k$
is a hyper-parameter and $\sum_{j=1}^{k}p_{j}({\bm{x}})=1$ for all ${\bm{x}}$.
We then introduce $k$ affine fields
${\bm{u}}_{j}({\bm{x}})={\bm{A}}_{j}{\bm{x}}+{\bm{b}}_{j}$, $j\in[k]$, and
define our deformation loss as:

$\mathrm{loss}_{\text{d}}(\theta)=\min_{{\bm{u}}_{1}\ldots{\bm{u}}_{k}}\frac{1}{n}\sum_{j=1}^{k}\sum_{i=1}^{n}p_{j}({\bm{x}}_{i})\rho({\bm{x}}_{i};{\bm{v}}_{{\bm{z}}_{i},{\bm{\eta}}_{i},{\bm{u}}_{j}})$
(12)

This loss divides the surface ${\mathcal{S}}({\bm{x}})$ to (soft) pieces
according to the probability vector ${\bm{p}}$, and encourages each piece to
be deformed with a rigid motion. Minimization w.r.t.
${\bm{u}}_{1},\ldots,{\bm{u}}_{k}$ can be done independently for each
$j\in[k]$, as in the case of a single rigid deformation prior. The only
difference is the incorporation of weights $p_{j}({\bm{x}}_{i})$, leading to a
weighted LS problem. Using Danskin’s theorem again will guarantee that
plugging the minimizing fields ${\bm{A}}_{j}^{\star},{\bm{b}}^{\star}_{j}$ in
the loss in eq. 12 provides correct value and gradient at current parameter
$\theta$.

##  4 Implementation details

In addition to the deformation prior $\mathrm{loss}_{\text{d}}$, which is the
main contribution of this work and described in the previous section, we use
standard reconstruction losses for the shape space requirement (i), _i.e_.,
${\mathcal{S}}({\bm{z}}^{(i)})\approx{\mathcal{X}}^{(i)}$, $i\in[m]$. In this
section we provide the remaining details on our loss and architecture.

We use two networks in our system, the implicit representation
$f:\mathbb{R}^{3}\times\mathbb{R}^{D}\rightarrow\mathbb{R}$, and the
probability network
${\bm{p}}=(p_{1},\ldots,p_{k}):\mathbb{R}^{3}\times\mathbb{R}^{D}\rightarrow\mathbb{R}^{k}$.
We use $k=20$ unless otherwise stated, an ablation test on $k$ is provided in
the supplementary. Both networks are Multilayer Perceptrons (MLPs) (specific
architectures are in the supplementary). ${\bm{p}}({\bm{x}},{\bm{z}})$
represents the parts’ probabilities of a point ${\bm{x}}$ in the shape
${\mathcal{S}}({\bm{z}})$. We use the Auto-Decoder (AD) paradigm [31], where
the latent codes ${\mathcal{Z}}=\left\\{{\bm{z}}^{(i)}\right\\}_{i\in[m]}$ are
learnable parameters. We incorporate the AD regularizer encouraging the latent
codes to have a scaled Gaussian distribution (see [31] for more details),

$\mathrm{loss}_{\text{ad}}(\theta)=\frac{1}{m}\sum_{i=1}^{m}\|{\bm{z}}^{(i)}\|^{2},$
(13)

where $\left\|{\bm{z}}\right\|=\sqrt{{\bm{z}}^{T}{\bm{z}}}$ is the standard
euclidean norm, and we henceforth denote by $\theta$ the collection of all
learnable parameters of the system, including the parameters of $f$,
${\bm{p}}$, and ${\mathcal{Z}}$.

##### Deformation loss.

We use eq. 12 as our deformation loss, where the samples
${\bm{z}}_{i},{\bm{\eta}}_{i},{\bm{x}}_{i}$ used for each iteration are
constructed as follows. For each $i\in[n]$ in a batch of size $n$, we random a
pair of latent codes ${\bm{z}}^{(i_{1})},{\bm{z}}^{(i_{2})}$ from
${\mathcal{Z}}$, and a scalar $t$ uniformly in $[0,1]$. Next, we define
${\bm{z}}_{i}$ to be the time $t$ interpolation of ${\bm{z}}^{(i_{1})}$ and
${\bm{z}}^{(i_{2})}$. We have tested two options: _linear_ , where
${\bm{z}}_{i}=(1-t){\bm{z}}^{(i_{1})}+t{\bm{z}}^{(i_{2})}$, and _spiral_ ,
where

${\bm{z}}_{i}=\left[(1-t)\|{\bm{z}}^{(i_{1})}\|+t\|{\bm{z}}^{(i_{2})}\|\right]\text{slerp}\left(\frac{{\bm{z}}^{(i_{1})}}{\|{\bm{z}}^{(i_{1})}\|},\frac{{\bm{z}}^{(i_{2})}}{\|{\bm{z}}^{(i_{2})}\|}\right)$

and slerp is spherical linear interpolation. We found the spiral interpolation
favorable since it does not decrease the interpolated latent codes norm; we
compare between the two options in the experimental section. For each such
${\bm{z}}_{i}$, we set ${\bm{\eta}}_{i}$ to be the normalized speed
$\frac{d{\bm{z}}_{i}}{dt}$ of the interpolation, and ${\bm{x}}_{i}$ a random
sample from ${\mathcal{S}}({\bm{z}}_{i})$ computed by projecting a random
point in space onto ${\mathcal{S}}({\bm{z}}_{i})$ as detailed in the
supplementary. We also add the Eikonal loss [18], denoted
$\mathrm{loss}_{\text{e}}$, to regularize the level-sets of implicit surfaces
at intermediate latent codes ${\bm{z}}_{i}$, see supplementary for details.

|
---|---
|
Figure 4: Toy dataset: incorporating the deformation prior
$\mathrm{loss}_{\text{d}}$ (top row) allows reconstructing natural shape
interpolation, in contrast to baseline (bottom row). Left and right in each
sequence are the reconstructed input shapes.

##### Reconstruction loss.

In order to approximate the input shapes ${\mathcal{X}}^{(i)}$ at the latent
codes ${\bm{z}}^{(i)}$, we use the SALD reconstruction loss [4], denoted
$\mathrm{loss}_{\text{r}}$, that handles raw data as input, and only requires
the _unsigned_ distance to the input geometry,
$d({\bm{x}},{\mathcal{X}})=\min_{{\bm{y}}\in{\mathcal{X}}}\left\|{\bm{x}}-{\bm{y}}\right\|$.
See supplementary for details on this loss. Our complete loss function is

$\mathrm{loss}=\mathrm{loss}_{\text{r}}+\lambda_{\text{d}}\mathrm{loss}_{\text{d}}+\lambda_{\text{e}}\mathrm{loss}_{\text{e}}+\lambda_{\text{ad}}\mathrm{loss}_{\text{ad}},$
(14)

where the losses are all functions of the parameters $\theta$, and we use
$\lambda_{\text{e}}=0.1$, and $\lambda_{\text{ad}}=0.001$. For
$\lambda_{\text{d}}$ we use scheduling, details in the supplementary.

##  5 Experiments

##### Baselines.

For baselines we picked two popular implicit shape space learning methods that
have been shown to produce state of the art results on the same or similar
datasets: (i) Auto-Decoder (AD) [31]; and (ii) Variational Auto-Encoder (VAE)
[25]. In both cases, we trained using the SALD reconstruction loss [4]. For
the encoder in the VAE, we use PointNet [33], similar to previous works that
utilizing VAE on this task [25, 3, 4]. The full details of the network
architecture can be found in supplementary.

###  5.1 Evaluation using toy data

In the first experiment, we test the ability of our model to learn from a
synthetic shape space with known and well understood deformations between
shapes. To this end, we generated two “toy” datasets: (i) 12 cubes, randomly
rotated and translated in 3D space; and (ii) 12 ellipsoids (“main body”) with
upper and bottom perturbing part (“arms” and “legs”), where the main body
ellipsoid is randomly translated in the $x$-axis direction and the arms and
legs ellipsoids are deformed by two different random rotations around the
$z$-axis. For each pair of shapes in this dataset there exists a two-piece
rigid deformation field, deforming one shape to the other. See the
supplementary for a video showing few samples of shapes from the two datasets.

On each of these datasets we trained: (i) Vanilla auto-decoder, _i.e_.,
without our deformation prior loss, $\lambda_{\text{d}}=0$; and (ii) using the
same auto-decoder architecture and loss as in (i), but now with active
deformation loss, _i.e_., $\lambda_{d}=0.001$. We set $k$ to 1 for the cubes
and 2 for the ellipses to match the ground truth number of parts.

The bottom and upper row in fig. 4 show learned implicit shapes for
$\lambda_{d}=0$ and $\lambda_{d}=0.001$, respectively. In each sequence, the
leftmost and rightmost shapes correspond to a pair of learned latent codes
${\bm{z}}^{(i_{1})},{\bm{z}}^{(i_{2})}$ , $i_{1},i_{2}\in[12]$. Note that both
methods learn to reconstruct the input data well. The middle shapes in this
experiment were generated using linear interpolation of latent codes,
$(1-t){\bm{z}}^{(i_{1})}+t{\bm{z}}^{(i_{2})}$,
$t\in\left[0.15,0.35,0.65,0.85\right]$. Note, in particular, that the shapes
generated with $\lambda_{d}=0$ fail to keep important geometrical features and
revert to smooth but “non-geometrical” interpolation. This can be seen from
the smoothing of the cubes and the blending of the arms and legs in the
ellipsoid. In contrast, enabling the deformation loss leads to geometrically-
plausible interpolations, close to the ground-truth ones.

###  5.2 Shape space learning

Next, we test our deformation prior on the task of learning a shape space from
“real-life” raw scans. We considered the D-Faust raw scans dataset [8],
consisting of 41k scans from 10 humans: 5 male and 5 female. The data for each
individual is categorized into sequences of actions. Together with the raw
data, mesh registrations are also in the dataset. We use the registrations for
evaluation, but not for training.

Registrations Scans Method Chamfer Wasserstein Chamfer Wasserstein Unique
action Ours 0.112 2.994 0.098 2.971 AD 0.231 3.527 0.195 3.354 VAE 0.237 3.361
0.146 3.226 Punching Ours 0.126 3.164 0.103 3.069 AD 0.995 3.757 0.198 3.671
VAE 0.656 4.558 0.613 4.351 One leg jump Ours 0.278 3.490 1.313 5.601 AD 0.385
3.961 1.414 5.908 VAE 0.957 5.077 1.992 7.064 Light hopping Ours 0.053 2.801
0.045 2.871 AD 0.050 2.818 0.041 2.857 VAE 0.086 2.906 0.081 2.903 Shake hips
Ours 0.097 3.016 0.211 3.358 AD 0.128 3.228 0.244 3.532 VAE 0.217 3.258 0.343
3.469
---
Table 1: Reconstruction of test set of all humans (splits 1 and 2) in D-Faust.
We log Chamfer and Wasserstein distances of the reconstructed surfaces to the
raw scans and ground-truth registrations; Chamfer reported $*10^{3}$.
Registrations Scans Method Chamfer Wasserstein Chamfer Wasserstein Male random
Ours (spiral) 0.073 3.329 0.048 3.233 Ours (linear) 0.080 3.360 0.051 3.328 AD
0.071 3.361 0.045 3.292 VAE 0.128 3.503 0.164 3.457 Female random Ours
(spiral) 0.042 2.490 0.042 2.530 Ours (linear) 0.036 2.450 0.036 2.470 AD
0.056 2.600 0.049 2.640 VAE 0.163 2.841 0.152 2.842 Male action Ours (spiral)
1.017 6.410 0.822 5.360 Ours (linear) 0.762 5.499 0.649 4.616 AD 1.201 6.657
0.979 5.613 VAE 2.443 9.171 2.270 8.182 Female action Ours (spiral) 0.048
2.518 0.042 2.456 Ours (linear) 0.052 2.633 0.045 2.620 AD 0.068 2.806 0.059
2.709 VAE 8.509 17.024 8.547 16.951
---
Table 2: Reconstruction of test set of individuals (splits 3 and 4) in
D-Faust. We log the Chamfer and Wasserstein distances of the reconstructed
surfaces to the raw scans and ground-truth registrations; Chamfer reported
$*10^{3}$.

##### Shape space reconstruction.

To test the ability of our learned shape space to generalize and represent
unseen geometries from sparse training examples, we have created the following
data splits from the D-Faust dataset: (1) _Unique action_ : removes a unique
action from each of the 10 human identities as a test set, and trains on every
5-th scan from the remaining time-ordered scans; (2) _Punching_ /_One leg
jump_ /_Light hopping_ /_Shake hips_ : removes the same action from all 10
humans as a test set, and trains on every 5th scan of the time-ordered
remainder. (3) _Male action_ /_Female action_ : takes one action from each
human as a test set, and out of a tuple $\mathcal{F}$ of all remaining ordered
frames takes every 10-th for training. (4) _Male random_ /_Female random_ :
uses the same training set as (3), but from every 10 consecutive frames in
$\mathcal{F}$, randomly samples from frames 1-9 for testing. In general, since
all these data splits are still very large, we randomly subsampled 5%-10% of
the scans for experimentation.

To test how well the learned shape space represents unseen test cases, we
follow [31] and, for each test scan ${\mathcal{X}}$, we minimize the
reconstruction loss with respect to the latent code
${\bm{z}}\in\mathbb{R}^{D}$ with the weights of the neural network $f$ fixed.
Table 1 and table 2 report the results on splits 1 and 2 (all humans), and on
splits 3 and 4 (individuals) respectively. In the latter, we also provide a
comparison between linear and spiral interpolations. Figure 6 shows some
typical results from splits 1 and 2; we show failure cases in the
supplementary. In order to measure reconstruction accuracy we use the Chamfer
and Wasserstein distances, as customary (details in the supplementary). As can
be seen by inspecting the quantitative and qualitative results, our
deformation prior indeed improves prediction ability to unseen scans
representing a missing action or pose. For example, when comparing the red
boxes in Figure 6 to the blue boxes (unseen ground truth) our method is able
to reconstruct the most similar pose, outperforming the baselines. Figure 5
depicts the probability vectors (a different color for each of the $k=20$
pieces) for reconstructed examples of the Female (splits 3 and 4).
Interestingly, the vectors correspond to the natural kinematic structure of
the body.

Figure 5: The learned probabilities ${\bm{p}}$ for several reconstructions from the female split (splits 3 and 4). Figure 6: Shape space reconstruction of unseen poses (taken from splits 1,2 and 4). From left to right (in each sequence of 4): unseen raw scan, ours, AD, VAE. |
---|---
Figure 7: Interpolation of latent codes. Each block of three rows shows (top
to bottom): Our result, AD, and VAE. Note the natural shape and motion of the
intermediate shapes when incorporating our deformation prior.  Method
HKS($t_{\text{min}}$) HKS($t_{\text{mean}}$) HKS($t_{\text{max}}$) area
geodesic Male Ours (slerp) 955.422 4.804 4.775 120.632 0.717 Ours (linear)
775.381 4.946 4.914 179.856 0.977 AD 1004.734 6.150 6.104 189.676 1.119 Female
Ours (slerp) 191.823 6.511 6.433 106.807 0.836 Ours (linear) 239.745 7.664
7.584 143.439 1.073 AD 468.887 14.830 14.669 151.859 1.203
---
Table 3: Evaluation of interpolated surfaces. Wasserstein distance between
histograms of different surface properties; HKS numbers are $*10^{6}$; area
and geodesic $*10^{3}$.

###  5.3 Shape space interpolation

We evaluated shape space interpolations on the individual splits (splits 3 and
4). The interpolations are using the spiral interpolation of learned latent
codes ${\mathcal{Z}}$. Figure 7 presents qualitative results; note that our
deformation prior improves intermediate poses and alleviates shortening of
limbs that are common with the baselines. For example, note that the
highlighted face, hand, and legs in the red boxes are either distorted or
deformed in the baselines compared to the interpolation end points (blue
boxes), while our method maintains the geometric properties of these parts.
For qualitative assessment of the interpolation results, we have compared
histograms of different surface properties such as Heat Kernel Signature (HKS)
at times $t_{\text{min}}$,
$t_{\text{mean}}=0.5(t_{\text{min}}+t_{\text{max}}$), and $t_{\text{max}}$ as
suggested in [40], surface area, and pairwise geodesic distances [30]. In
Table 3 we log the Wasserestein distances between histograms of interpolated
surfaces and the corresponding histograms of the reconstructed shapes
generated the interpolation (_i.e_., far left and right in each sequence in
Figure 7). Note the advantage in using spiral interpolation method.

##  6 Conclusions

In this paper we have analyzed the ability of deep neural implicit surfaces to
interpolate between members of a set of complex 3D shapes, such as articulated
human bodies. To this end, we proposed a novel framework that regularizes the
deformations within an implicit neural shape space. The latter is done by
first expressing the 3D deformation fields generated by an infinitesimal
change of a latent shape code, followed by regularizing these fields to be a
collection of approximately rigid motions with an as-rigid-as-possible
regularizer. The experimental section has demonstrated that our framework
significantly improves the generalization of the state of the art implicit
shape representation networks especially when trained on a sparse and
temporally non-smooth collection of 3D shapes, such as a heavily subsampled
version of the raw D-Faust dataset.

## References

  * [1] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas.  Learning representations and generative models for 3d point clouds.  In International conference on machine learning, pages 40–49. PMLR, 2018.
  * [2] Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israelov, Haggai Maron, and Yaron Lipman.  Controlling neural level sets.  In Advances in Neural Information Processing Systems, pages 2032–2041, 2019.
  * [3] Matan Atzmon and Yaron Lipman.  Sal: Sign agnostic learning of shapes from raw data.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [4] Matan Atzmon and Yaron Lipman.  Sald: Sign agnostic learning with derivatives.  arXiv preprint arXiv:2006.05400, 2020.
  * [5] Tamer Başar and Pierre Bernhard.  H-infinity optimal control and related minimax design problems: a dynamic game approach.  Springer Science & Business Media, 2008.
  * [6] Mirela Ben-Chen, Adrian Butscher, Justin Solomon, and Leonidas Guibas.  On discrete killing vector fields and patterns on surfaces.  In Computer Graphics Forum, volume 29, pages 1701–1711. Wiley Online Library, 2010.
  * [7] Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  Dynamic FAUST: Registering human bodies in motion.  In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [8] Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  Dynamic FAUST: registering human bodies in motion.  In Proc. CVPR, 2017.
  * [9] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5939–5948, 2019.
  * [10] Julian Chibane, Gerard Pons-Moll, et al.  Neural unsigned distance fields for implicit function learning.  Advances in Neural Information Processing Systems, 33, 2020.
  * [11] Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese.  3d-r2n2: A unified approach for single and multi-view 3d object reconstruction.  In European conference on computer vision, pages 628–644. Springer, 2016.
  * [12] Luca Cosmo, Antonio Norelli, Oshri Halimi, Ron Kimmel, and Emanuele Rodolà.  Limp: Learning latent shape representations with metric preservation priors.  In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III, volume 12348, pages 19–35. Springer, 2020.
  * [13] Marvin Eisenberger and Daniel Cremers.  Hamiltonian dynamics for real-world shape interpolation.  In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV, volume 12349, pages 179–196. Springer, 2020.
  * [14] Marvin Eisenberger, Zorah Lähner, and Daniel Cremers.  Divergence-free shape correspondence by deformation.  In Computer Graphics Forum, volume 38, pages 1–12. Wiley Online Library, 2019.
  * [15] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser.  Local deep implicit functions for 3d shape.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4857–4866, 2020.
  * [16] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser.  Learning shape templates with structured implicit functions.  In Proceedings of the IEEE International Conference on Computer Vision, pages 7154–7164, 2019.
  * [17] Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta.  Learning a predictable and generative vector representation for objects.  In European Conference on Computer Vision, pages 484–499. Springer, 2016.
  * [18] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman.  Implicit geometric regularization for learning shapes.  In Proceedings of Machine Learning and Systems 2020, 2020.
  * [19] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry.  3d-coded: 3d correspondences by deep deformation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 230–246, 2018.
  * [20] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry.  A papier-mâché approach to learning 3d surface generation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 216–224, 2018.
  * [21] Chiyu Jiang, Jingwei Huang, Andrea Tagliasacchi, Leonidas Guibas, et al.  Shapeflow: Learnable deformations among 3d shapes.  arXiv preprint arXiv:2006.07982, 2020.
  * [22] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [23] Lingxiao Li, Minhyuk Sung, Anastasia Dubrovina, Li Yi, and Leonidas J Guibas.  Supervised fitting of geometric primitives to 3d point clouds.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2652–2660, 2019.
  * [24] Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui Wang.  3d shape reconstruction from sketches via multi-view convolutional networks.  In 2017 International Conference on 3D Vision (3DV), pages 67–77. IEEE, 2017.
  * [25] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4460–4470, 2019.
  * [26] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing scenes as neural radiance fields for view synthesis.  In Proc. ECCV, 2020.
  * [27] Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon.  Kinectfusion: Real-time dense surface mapping and tracking.  In 2011 10th IEEE International Symposium on Mixed and Augmented Reality, pages 127–136. IEEE, 2011.
  * [28] Michael Niemeyer, Lars M. Mescheder, Michael Oechsle, and Andreas Geiger.  Occupancy flow: 4d reconstruction by learning particle dynamics.  In Proc. ICCV, 2019.
  * [29] David Novotny, Diane Larlus, and Andrea Vedaldi.  Learning 3d object categories by looking around them.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5218–5227, 2017.
  * [30] Robert Osada, Thomas Funkhouser, Bernard Chazelle, and David Dobkin.  Shape distributions.  ACM Transactions on Graphics (TOG), 21(4):807–832, 2002.
  * [31] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [32] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * [33] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 652–660, 2017.
  * [34] Stephan R Richter and Stefan Roth.  Matryoshka networks: Predicting 3d geometry via nested shape layers.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1936–1944, 2018.
  * [35] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  Advances in Neural Information Processing Systems, 33, 2020.
  * [36] Miroslava Slavcheva, Maximilian Baust, Daniel Cremers, and Slobodan Ilic.  Killingfusion: Non-rigid 3d reconstruction without correspondences.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1386–1395, 2017.
  * [37] Justin Solomon, Mirela Ben-Chen, Adrian Butscher, and Leonidas J. Guibas.  As-killing-as-possible vector fields for planar deformation.  Comput. Graph. Forum, 30(5), 2011.
  * [38] Olga Sorkine and Marc Alexa.  As-rigid-as-possible surface modeling.  In Symposium on Geometry processing, volume 4, pages 109–116, 2007\.
  * [39] Jos Stam and Ryan Schmidt.  On the velocity of an implicit surface.  ACM Transactions on Graphics (TOG), 30(3):1–7, 2011.
  * [40] Jian Sun, Maks Ovsjanikov, and Leonidas Guibas.  A concise and provably informative multi-scale signature based on heat diffusion.  In Computer graphics forum, volume 28, pages 1383–1392. Wiley Online Library, 2009.
  * [41] Michael Tao, Justin Solomon, and Adrian Butscher.  Near-isometric level set tracking.  In Computer Graphics Forum, volume 35, pages 65–77. Wiley Online Library, 2016.
  * [42] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox.  Multi-view 3d models from single images with a convolutional network.  In European Conference on Computer Vision, pages 322–337. Springer, 2016.
  * [43] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox.  Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2088–2096, 2017.
  * [44] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang.  Pixel2mesh: Generating 3d mesh models from single rgb images.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 52–67, 2018.
  * [45] Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann.  3dn: 3d deformation network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1038–1046, 2019.
  * [46] Francis Williams, Jerome Parent-Levesque, Derek Nowrouzezahrai, Daniele Panozzo, Kwang Moo Yi, and Andrea Tagliasacchi.  Voronoinet: General functional approximators with local support.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 264–265, 2020.
  * [47] Francis Williams, Teseo Schneider, Claudio Silva, Denis Zorin, Joan Bruna, and Daniele Panozzo.  Deep geometric prior for surface reconstruction.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10130–10139, 2019.
  * [48] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum.  Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling.  In Advances in neural information processing systems, pages 82–90, 2016.
  * [49] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann.  Disn: Deep implicit surface network for high-quality single-view 3d reconstruction.  In Advances in Neural Information Processing Systems, pages 492–502, 2019.

##  7 Appendix

Figure 8: Failures in reconstructing unseen poses. From left to right (in each sequence of 4): unseen raw scan, ours, AD, VAE. |
---|---
$k=1$ | $k=5$
|
$k=10$ | $k=15$
|
$k=40$ | $k=200$
|
$k=400$ | $k=\left|{\mathcal{X}}\right|$
Figure 9: Ablation: experimenting with various $k$ values, the number of
affine fields. Left and right in each row are the learned reconstructions. The
middle shapes were generated by latent interpolation.

###  7.1 Additional Results

We provide a video file named toy_figure.mp4 showing animation sequences of
results and data used for the evaluation in Section 5.1. In addition, we
provide video files male.mp4 and female.mp4, showing animation sequences of
learned interpolations from split 3 in the experiment described in Section
5.2. For splits 1 and 2, qualitative examples of interpolations between
different humans are shown in Figure 11.

###  7.2 Ablation over $k$, the number of affine fields

The main parameter of our method is $k$, the number of affine fields (see
equation 7) in our multiple deformation prior loss. The intuition in setting
$k$ is that it should capture the number of different parts of a shape that
move rigidly. To test the effect of $k$ we designed the following experiment.
We considered two input shapes ${\mathcal{X}}_{1},{\mathcal{X}}_{2}$ and
trained using our loss as in equation 14, under different number values for
$k$. In each run, all the other hyper-parameters were set to be the same:
$\lambda_{\text{e}}=0.1$, $\lambda_{\text{ad}}=0.001$,
$\lambda_{\text{d}}=0.001$ and $D=1$. Figure 9 shows learned input
reconstructions (far left and right); and in addition, it shows shapes
generated by linear interpolation in the one dimensional latent space. Our
findings can be summarized as follows: the number $k$ serves merely as an
upper bound to the number of deforming rigid parts that can be explained by
the deformation prior loss. That is, $k$ should not be set too small with
respect to the expected number of independent deforming rigid parts, but other
than that there is no harm in setting $k$ to be much bigger. In figure 9 we
see comparable results in quality for
$k\in\left\\{5,10,15,40,200,400\right\\}$. Note that setting $k$ too small, as
in $k=1$, result in poor quality reconstructions. One question that might
raise from these results is why should we limit the number of affine fields,
$k$, at all? That is, setting $k=\left|{\mathcal{X}}\right|$ by allowing a
separate rigid deformation prior for each $x\in{\mathcal{X}}$. We see that for
$k=\left|{\mathcal{X}}\right|$, we get high quality reconstruction as well.
Nonetheless, deformations in latent space are no longer seem to be natural and
fail to capture multiple parts rigid deformation between the two input shapes.
This may be attributed to the fact that using a different, non-consistent
affine field at every point allows too much flexibility and can reduce the
killing loss without resulting in a near-rigid deformation.

###  7.3 Failure cases

Figure 8 shows some typical failures of our method. Left part depicts a test
case from the removed punching sequence in split 2, whereas the right part is
from the removed_one_leg_jump sequence in split 1. In essence, although we
demonstrate improvement over baselines there are still cases where the network
fails to generalize to unseen poses. See the bend in the right arm in the left
example and the missing parts from right hand in the right example. One
possible solution could be to train our loss not only between pairs of latents
in ${\mathcal{Z}}$, but also to more general latent samples in
$\mathbb{R}^{D}$ to allow better extrapolation.

###  7.4 Additional Implementation details

Figure 10: Timings. We report the time of a single forward pass (in seconds)
as a function of sample size.

##### Architecture.

Our architecture consists of two networks
$f:\mathbb{R}^{3}\times\mathbb{R}^{D}\rightarrow\mathbb{R}$, and the
probability network
${\bm{p}}=(p_{1},\ldots,p_{k}):\mathbb{R}^{3}\times\mathbb{R}^{D}\rightarrow\mathbb{R}^{k}$.
In all experiements in the paper we used $D=256,k=20$, unless stated
otherwise. The network ${\bm{p}}$ is implemented by a ReLU activation one-
layer MLP with 128 hidden units. The network $f$ is implemented by a 8-layers
MLP, with a single skip connection between the input to the middle layer.
There are $512$ units in each hidden layer. Note that the same architecture
was used in [31, 3, 18, 4]. For the initialization of $f$ weights, we use the
geometric initialization from [3]. We use the SoftPlus function, with
$\beta=100$ for activation. The same architecture is used for the AutoDecoder
baseline and for the decoder in the VAE baseline. For the encoder in the VAE
baseline we use the same one as in [4].

Next, we describe additional details regarding the terms that consists our
loss, see equation 14 in the main paper.

##### Eikonal loss.

We utilize the Eikonal loss [18] to regularize the level-sets of implicit
surfaces at intermediate latent codes ${\bm{z}}_{i}$

$\mathrm{loss}_{\text{e}}(\theta)=\frac{1}{n}\sum_{i=1}^{n}\left(\left\|\nabla_{\bm{x}}f({\bm{y}}_{i},{\bm{z}}_{i})\right\|-1\right)^{2},$
(15)

where ${\bm{y}}_{i}\in\mathbb{R}^{3}$ are sampled uniformly in a bounding box
of all the input shapes.

##### Deformation loss.

Here we give the missing details from the main text regarding the sampling of
the points ${\bm{x}}_{i}\in{\mathcal{S}}({\bm{z}}_{i})$. We start by drawing a
uniform sample of points ${\bm{p}}_{i}$ in a bounding box of the shape
${\mathcal{S}}({\bm{z}}_{i})$. Then, we perform $5$ iterations of the
generalized Newton, defined by:

${\bm{p}}_{i}^{\text{next}}={\bm{p}}_{i}-\frac{\nabla_{{\bm{x}}}f^{T}}{\left\|\nabla_{{\bm{x}}}f\right\|^{2}}f({\bm{p}}_{i}).$
(16)

Finally, we set ${\bm{x}}_{i}={\bm{p}}_{i}^{\text{next}}+{\bm{n}}_{i}$, where
${\bm{n}}_{i}$ is a random sample from a Gaussian noise
${\mathcal{N}}(0,0.02)$.

##### Reconstruction loss.

In order to approximate the input shapes ${\mathcal{X}}^{(i)}$ at the latent
codes ${\bm{z}}^{(i)}$, we use the SALD reconstruction loss [4] that handles
raw data as input, and only requires the _unsigned_ distance to the input
geometry,
$d({\bm{x}},{\mathcal{X}})=\min_{{\bm{y}}\in{\mathcal{X}}}\left\|{\bm{x}}-{\bm{y}}\right\|$.
For a batch of size $b$ of samples ${\bm{q}}_{i},{\bm{z}}^{(i)}$, $i\in[b]$,
the loss is defined by

$\displaystyle\mathrm{loss}_{\text{r}}(\theta)$
$\displaystyle=\frac{1}{b}\sum_{i=1}^{b}\Big{[}\tau\big{(}f({\bm{q}}_{i},{\bm{z}}^{(i)}),d({\bm{q}}_{i},{\mathcal{X}}^{(i)})\big{)}$
$\displaystyle+\lambda\tau\big{(}\nabla_{\bm{x}}f({\bm{q}}_{i},{\bm{z}}^{(i)}),\nabla_{\bm{x}}d({\bm{q}}_{i},{\mathcal{X}}^{(i)})\big{)}\Big{]},$

where $\tau$ is the sign agnostic function, _i.e_.,
$\tau(a,b)=\min\left\\{|a-b|,|a+b|\right\\}$ for scalars, and
$\tau({\bm{a}},{\bm{b}})=\min\left\\{\left\|{\bm{a}}-{\bm{b}}\right\|,\left\|{\bm{a}}+{\bm{b}}\right\|\right\\}$
for vectors. The latent ${\bm{z}}^{(i)}$ is a random sample from
${\mathcal{Z}}$, and the point ${\bm{q}}_{i}$ is sampled by adding random
displacement, ${\bm{n}}_{i}$, to a uniform sample from the input geometry
${\bm{x}}_{i}\in{\mathcal{X}}^{(i)}$. That is,
${\bm{q}}_{i}={\bm{x}}_{i}+{\bm{n}}_{i}$. The displacement distribution is a
mixture of two isotropic Gaussians, ${\mathcal{N}}(0,\sigma_{i,1}^{2}I)$ and
${\mathcal{N}}_{i}(0,\sigma_{i,2}^{2}I)$. The parameter $\sigma_{i,1}^{2}$
depends on each ${\bm{x}}_{i}$ and is set to be the distance of the 10 closest
point to ${\bm{x}}_{i}$, whereas $\sigma_{i,2}$ was set to $0.3$ fixed.
Lastly, in all experiments we set $\lambda=0.1$.

####  7.4.1 Training details

In all the experiments in section 5.2, including baselines, training was done
using the Adam [22] optimizer, with batch size $b=64$. For the VAE, we set a
fixed learning rate of $0.0005$. The same learning rate was set for the AD,
except for the learning rate of the latents $\left\\{{\bm{z}}^{(i)}\right\\}$,
which was set to $0.001$. For splits 1 and 2, networks were trained for $5000$
epochs. The loss parameter $\lambda_{\text{d}}$ was set according to the
following scheduling: $\lambda_{\text{d}}=0$ for the first $2000$ epochs,
followed by $\lambda_{\text{d}}=0.001$ for the next $2000$ epochs, and
$\lambda_{\text{d}}=0.0001$ for the last $1000$ epochs. For splits 3 and 4,
training was done for 50k epochs. The scheduling for $\lambda_{\text{d}}$ was
to set $\lambda_{\text{d}}=0$ for the first 20k epochs, followed by
$\lambda_{\text{d}}=0.001$ for the next 20k epochs, and
$\lambda_{\text{d}}=0.0001$ for the last 10k epochs. Training was done on $4$
Nvidia V-100 GPUs, using pytorch deep learning framework [32]. For the
evaluation in section 5.1, training was for $5000$ epochs for all experiments,
on a single Nvidia V-100 GPU. The batch size was $b=8$. In addition, we set
$\lambda_{\text{d}}=0.001$ fixed.

##### Timings.

In figure 10 we report the total seconds required for a single forward pass,
on a single Nvidia V-100 GPU. The baseline for comparison is SALD [4]. In the
experiments in Section 5.2, a sample size of $2500$ points was used, resulted
in approximately $9.7$ times slower training than SALD. The main additional
cost in our method can be attributed to the calculation of the network’s
second derivatives, as part of the calculation of the rigid deformation prior.

####  7.4.2 Evaluation metrics

We used two different metrics to measure distances between shapes. One metric
is the _Chamfer_ distance, $\textrm{d}_{\text{C}}$, measuring the squared
distance between each point in one shape to its nearest neighbour in the
other. Second metric is the _Wasserstein_ distance, $\textrm{d}_{\text{W}}$,
which measures the sum of the distances of the optimal transportation between
the shapes. See [1] for more details about these two metrics. Next, we present
the formal definition for these metrics. The definition of the _Chamfer_
distance is

$\displaystyle\textrm{d}_{\text{C}}\left({\mathcal{X}}_{1},{\mathcal{X}}_{2}\right)$
$\displaystyle=\frac{1}{2}\left(\textrm{d}_{\text{C}}^{\rightarrow}\left({\mathcal{X}}_{1},{\mathcal{X}}_{2}\right)+\textrm{d}_{\text{C}}^{\rightarrow}\left({\mathcal{X}}_{2},{\mathcal{X}}_{1}\right)\right)$
(17)

where

$\displaystyle\textrm{d}_{\text{C}}^{\rightarrow}\left({\mathcal{X}}_{1},{\mathcal{X}}_{2}\right)$
$\displaystyle=\frac{1}{\left|{\mathcal{X}}_{1}\right|}\sum_{{\bm{x}}_{1}\in{\mathcal{X}}_{1}}\min_{{\bm{x}}_{2}\in{\mathcal{X}}_{2}}\left\|{\bm{x}}_{1}-{\bm{x}}_{2}\right\|^{2}.$
(18)

The definition for the _Wasserstein_ distance is

$\displaystyle\textrm{d}_{\text{W}}\left({\mathcal{X}}_{1},{\mathcal{X}}_{2}\right)$
$\displaystyle=\min_{\phi:{\mathcal{X}}_{1}\rightarrow{\mathcal{X}}_{2}}\sum_{x\in{\mathcal{X}}_{1}}\left\|\phi\left(x\right)-x\right\|$
(19)

where $\phi$ is a bijection. Note that $\textrm{d}_{\text{W}}$ inputs of equal
size. In tables 1 and 2 we evaluated $\textrm{d}_{\text{C}}$ on samples of
size 30k points, and $\textrm{d}_{\text{W}}$ on samples of size 1k points.

####  7.4.3 Danskin’s Theorem

######  Theorem 1 (Danskin).

Let $g:\mathbb{R}\times\Omega\rightarrow\mathbb{R}$ be a continuous function,
continuously differentiable in its first argument, where
$\Omega\subset\mathbb{R}^{d}$ is a compact set. Let

$G(t,{\bm{x}})=\min_{{\bm{x}}\in\Omega}g(t,{\bm{x}}).$ (20)

Then

$\frac{\partial G}{\partial t}(t,{\bm{x}})=\frac{\partial G}{\partial
t}(t,{\bm{x}}_{\star}),$

when ${\bm{x}}_{\star}$ is the unique minimizer of eq. 20.

This version of Danskin’s Theorem follows directly from Corollary 10.1 in [5].

Figure 11: Interpolation of latent codes between diffent humans. Each block of
three rows shows (top to bottom): Our result, AD, and VAE.