# Deformable Surface Reconstruction via Riemannian Metric Preservation

Oriol Barbany† Adrià Colomé Carme Torras
Institut de Robòtica i Informàtica Industrial (CSIC-UPC), Barcelona, Spain

###### Abstract

Estimating the pose of an object from a monocular image is an inverse problem
fundamental in computer vision. The ill-posed nature of this problem requires
incorporating deformation priors to solve it. In practice, many materials do
not perceptibly shrink or extend when manipulated, constituting a powerful and
well-known prior. Mathematically, this translates to the preservation of the
Riemannian metric. Neural networks offer the perfect playground to solve the
surface reconstruction problem as they can approximate surfaces with arbitrary
precision and allow the computation of differential geometry quantities. This
paper presents an approach to inferring continuous deformable surfaces from a
sequence of images, which is benchmarked against several techniques and
obtains state-of-the-art performance without the need for offline training.

††$\dagger$ Corresponding author<EMAIL_ADDRESS>Figure 1: Summary of
the presented method. Given a template consisting of a surface representation
(in this case, a mesh) and an image showing such a surface, we want to recover
deformations of the same surface as it appears in the input frames. We propose
describing the surface by a parametric equation learned by an iterative
process. A parametric equation maps a point in $\mathbb{R}^{2}$, uniquely
defining a point on the surface, to its spatial coordinates in
$\mathbb{R}^{3}$ (in the example shown above, this mapping preserves the
colors, which we add for easier visualization). To find such a parametric
equation, we impose three soft constraints that alone are insufficient but,
when combined, allow us to recover the surface. We illustrate the ambiguities
of each condition by showing feasible solutions that are either correct (in
red) or incorrect (in blue and green). The constraints and ambiguities are:
(1) The surface correctly projects to the input image, a condition satisfied
by any surface obtained after arbitrarily moving the tracked points along the
line of sight. (2) The geodesic distances are preserved, a condition satisfied
for any isometric transformation of the template surface. (3) The surface does
not vary too much in consecutive frames, which is compatible with all surfaces
whose distance between the surface from the previous frames is small.

##  1 Introduction

In this paper, we tackle the problem of inferring the shape of a generic non-
rigid object from a sequence of monocular images given a 3D template of the
manipulated object. This problem is known as Shape from Template (SfT) [5] and
appears in many areas like entertainment [33], medicine [23], and robotic
manipulation [9]. Our motivation stems from this last field, in which
perceiving the state of deformable objects is one of the bottlenecks often
pinpointed as hindering their manipulation by robots [46]. In this area,
estimating the cloth state is crucial to simulate its evolution under
manipulation [13] and apply model-predictive control [26] to guide the robots.

The SfT problem is inherently ill-posed as it permits infinitely many
solutions leading to accurate projection to the input 2D images [48] and
requires the incorporation of deformation priors. One of the most common
hypotheses is that the object is obtained from an isometric transformation of
the template [5, 4, 34, 12, 10, 38, 11], meaning that it cannot extend or
shrink. This condition is often not restrictive in practice as many deformable
objects made of materials such as cloth and paper are nearly inextensible
[39]. Although this constitutes a powerful and widely applicable prior, it
defines a differential equation that, in most cases, has to be approximated to
make the problem computationally tractable.

In this work, we propose to encode the parametric equations of the object of
interest in the weights of a neural network. Parametric equations map 2D
coordinates representing a point on a surface to its 3D coordinates.
Therefore, they provide a continuous surface contrasting with the discrete
representations used in previous works. To find the parameters of the
parametric equations, we perform an iterative procedure enforcing three soft
constraints accounting for a correct surface projection, the preservation of
the surface metric (equivalent to the isometry constraint), and temporal
consistency with the previous surface. We depict the presented method in Fig.
1.

The isometry assumption is equivalent to the hypothesis that the geodesics on
the surface may not change their length over time, which we can enforce by
preserving the Riemannian metric. Given the parametric equations, we can
analytically compute differential geometric quantities such as the metric
tensor. Unlike previous methods struggling to optimize non-convex objectives
and considering differential equations, using a neural network and stochastic
optimizers allows for estimating the parameters of the surface in a relatively
small amount of time. While there exist methods using explicit neural
representations that have been used for single-view reconstruction [18, 8],
this is the first method to use them without offline training and in
conjunction with the popular isometry constraint.

We evaluate the proposed method against five approaches and a baseline
introduced in this paper on the publicly available datasets Deformable Surface
Tracking (DeSurT) and Texture-less Deformable Surfaces (TDS). The quantitative
results show that our method achieves the lowest or the second-lowest mean
tracking error and standard deviation in all the tested sequences, showing its
effectiveness and robustness. Compared to other optimization-based approaches
considered in this work, our solution is the fastest, which confirms that one
can analytically enforce isometry and keep the computation feasible using the
proposed framework.

Our main contributions are:

  * •

We combine the powerful and widely applicable assumption of metric
preservation with the representation power of neural networks. In particular,
we learn a surface parametrization [18], that allows representing continuous
surfaces rather than discretized representations (_e.g_., point clouds,
voxels, or meshes).

  * •

We advocate for imposing metric preservation of the surface as a soft
constraint, which accounts for the fact that physical quantities are not
exactly preserved [1, 2, 42].

  * •

We learn the parameters of the neural explicit surface during inference and
without an offline training process. Therefore, the method requires neither a
dataset nor fine-tuning to new sequences, materials, or template
representations.

The rest of this paper is structured as follows. In Section 2, we review
previous approaches to the SfT problem. Then, we introduce the proposed method
in Section 3 and show its qualitative and quantitative performance in Section
4. Finally, we conclude the paper in Section 5.

##  2 Related Work

In this section, we review several approaches to the SfT problem and group
them under three umbrellas: the methods that use hand-crafted constraints to
define the manifold of surfaces, those that infer the deformation models from
data, and the approaches that combine analytical and data-driven models. Table
1 presents a summary of the discussed papers.

Method |  Works with non-planar template |  Models sharp folds |  Does not need dataset |  Temporal coherence |  Variable mesh resolution
---|---|---|---|---|---
Hand-crafted constraints (Section 2.1)
Iterative isometric surfaces [10] | ✗ | ✓ | ✓ | ✗ | ✗
Closed-form isometric surfaces (I) [5, 4] | ✓ | ✗ | ✓ | ✗ | ✗
Closed-form isometric surfaces (II) [34, 12, 11] | ✓ | ✓ | ✓ | ✗ | ✗
Iterative constant Euclid. [43] | ✗ | ✗ | ✗ | ✓ | ✗
Closed-form constant Euclid. [42] | ✗ | ✗ | ✗ | ✗ | ✗
Inextensible surfaces [38] | ✗ | ✓ | ✓ | ✗ | ✗
Dense registration [30] | ✗ | ✓ | ✓ | ✓ | ✗
Laplacian meshes [31, 52] | ✓ | ✓ | ✓ | ✓ | ✗
Edge orientation changes [40] | ✓ | ✓ | ✓ | ✓ | ✗
Vertex coordinate changes [54] | ✓ | ✓ | ✓ | ✓ | ✗
Data-based constraints (Section 2.2)
Latent space [39, 45, 49] | ✓ | ✓ | ✗ | ✗ | ✗
Neural network: Image to vertices/depth [7, 17, 16, 37] | ✓ | ✓ | ✗ | ✗ | ✗
Surface parametrization [18, 8] | ✓ | ✓ | ✗ | ✗ | ✓
Hybrid approaches (Section 2.3)
GP constant Euclid. [44] | ✗ | ✗ | ✗ | ✗ | ✗
Differentiable physics simulator and renderer [20] | ✓ | ✓ | ✓ | ✓ | ✗
Ours | ✓ | ✓ | ✓ | ✓ | ✓

Table 1: Summary of related work. Our method is the first to satisfy all the
listed desirable properties for the reconstruction of deformable surfaces.

###  2.1 Hand-crafted constraints

These methods leverage explicit properties of the tracked surface to determine
the manifold of possible shapes. Given that constraints are hand-crafted, we
can easily modify them, and their effects are well understood. However, some
constraints rarely describe the complex and non-linear physics that real
surfaces exhibit for large deformations [42], which would require designing
complex objectives that need specific knowledge of the surface [38, 39].

We can reconstruct a surface unambiguously if its Riemannian metric is assumed
to be preserved [4, Theorem 1]. Brunet _et al_. [10] exploited this property
proposing a method that requires planar templates. Several works [5, 4, 34,
12, 11] enforced metric preservation by relying on a differentiable warp
between the template and the images in a sequence. Such warp is undefined in
occluded areas, and its required differentiability is incompatible with sharp
folds.

Given the difficulty of enforcing surface metric preservation, some works
proposed to use relaxations using the Euclidean norm between vertices.
Enforcing equality constraints [42, 43] is incompatible with sharp folds, and
inequality constraints [38, 39, 45] are prone to vertex collapsing. To prevent
the latter, some works use the maximum depth heuristic, which does not
consider surface properties [4].

Laplacian meshes provide a framework for constraining the problem by reducing
the number of free parameters [31, 52]. However, this approach loses
resolution on the edges of the surface and yields 3D shape estimates that re-
project to the image but are not necessarily accurate.

Another typical assumption is that the surface does not vary too much in
neighboring frames, which is a mild assumption for high enough image rates and
is known as the short-baseline case in the SfT literature. Enforcing temporal
smoothness helps recover surfaces with severe deformations and reduces jitter.
We can reduce frame-to-frame flickering by minimizing the difference with the
previous time step [54], a window of past time steps [38], the second
derivative of the surface parameters [43], or the change of edge orientation
along time [40]. Other works leverage temporal smoothness to obtain good
initializations to the solution [30, 31, 20].

###  2.2 Data-based constraints

Data-based approaches learn the manifold of plausible surfaces using
statistical learning techniques. The drawback of this class of methods is that
they require a dataset of surface configurations, which ideally should contain
all the possible deformations and be representative enough of the dynamics of
the material. Listing all deformations is impossible for deformable objects
like a cloth, which we can arrange in infinitely many ways. An additional
limitation is that, for these methods, we may need a different dataset for
each type of material, surface, lighting conditions, and mesh resolution.

We can obtain the manifold of possible shapes using dimensionality reduction
techniques. Previous works have considered PCA [39, 42, 43], sparse Gaussian
Process Latent Variable Models (GP-LVMs) [45] and constrained latent variable
models [49]. The dimensionality of the latent space may depend on the material
[43], and the models can yield extensible surfaces even if the dataset only
consists of inextensible surfaces [43].

An increasingly popular approach is to use neural networks to either predict
the surface vertices from an image [7, 37] or predict the depth map and use it
to infer the coordinates of the points in the tracked surface [17, 16]. In
general, such methods require learning on large datasets, and in some cases,
they only allow prohibitively small mesh sizes, _e.g_., $10\times 10$ [37].
The methods relying on depth estimation only recover the visible points in an
image and require post-processing the resulting point cloud with As-Rigid-As-
Possible regularization [47].

An interesting approach is to learn the parametric equations of the surface
depicted in the input image [18, 8], which can generate continuous surfaces.
These approaches are trained on pairs of images and their point clouds and
hence are also object-specific.

###  2.3 Hybrid approaches

A promising research direction highlighted in the context of perception for
robotic cloth manipulation [53] is to combine analytical and data-driven
models. Salzmann _et al_. [44] used Gaussian Processes (GPs) implicitly
satisfying a set of quadratic equality constraints, which are overly
simplistic for sharply folding materials like clothes [49]. This method also
requires that all the training examples satisfy all the constraints.

Kairanda _et al_. [20] proposed integrating a differentiable physics simulator
and renderer. The drawbacks of this method are that it imposes hard
constraints through the physics simulator, requires 16-24 hours on a GPU to
process an image sequence, and uses a fixed resolution of around 300 vertices,
which prevents capturing fine wrinkles [20].

This work falls into this category, as we also propose leveraging simulation
equations devised to constrain the dynamics of meshes and use them to solve
the inverse problem. Concretely, we adapt the inextensible model for the
manipulation of textiles [13], which follows the isometry assumption.

##  3 Methodology

In this section, we first introduce the problem notation and some
preliminaries. Then, we integrate the relaxation of the isometry constraint
developed for a cloth simulator [13] into a classical optimization scheme.
This method is used as a baseline and dubbed as Classical. Finally, we
introduce our method.

###  3.1 Preliminaries

The first fundamental form of a surface $\mathcal{S}$ allows measuring lengths
of curves, angles of tangent vectors, and areas of regions on it [14]. Using a
parametrization $\varphi:\mathcal{P}\subset\mathbb{R}^{2}\to\mathbb{R}^{3}$,
the Riemannian metric of $\mathcal{S}$ is then uniquely defined by the metric
tensor $\mathbf{J}_{\varphi}^{T}\mathbf{J}_{\varphi}$, where
$\mathbf{J}_{\varphi}$ is the Jacobian matrix of the map $\varphi$. We focus
on modeling surfaces in such a way that we preserve the Riemannian metric.
That is, at all times, the length of any curve inside the surface remains
constant.

######  Assumption 1.

The sequence of monocular images used as input to the SfT problem is obtained
with a calibrated camera with known intrinsic parameters.

Following the pinhole camera model, given a point
$\mathbf{s}=(x,y,z)\in\mathcal{S}$, which w.l.o.g. we assume to be expressed
in the camera referential, we can compute the position $(u,v)$ in the image
captured by the camera as follows:

$d\begin{bmatrix}u\\\ v\\\ 1\\\ \end{bmatrix}=\mathbf{K}\begin{bmatrix}x\\\
y\\\ z\end{bmatrix}\,,$ (1)

where $\mathbf{K}$ is the intrinsic matrix, known according to Assumption 1, a
common assumption in SfT, and $d$ is the depth along the line of sight. In the
following, let $\Pi(\mathbf{s}):=(u,v)$.

Problem: Given a sequence of images $\\{I^{(t)}\\}_{t\in[T]}$, where
$[T]:=\\{1,\dots,T\\}$, and a template surface $\mathcal{T}$, estimate the
surface $\mathcal{S}^{(t)}$ at any time $t$.

Figure 2: Problem notation. Scheme illustrating the notation used in this
manuscript. The parametric equations $\varphi^{(t)}$ map the input domain
$\mathcal{P}\in\mathbb{R}^{2}$, which in this case is the interior of a unit
square, to the 3D coordinates of $\mathcal{S}^{(t)}$, the surface at time $t$.
Then, an image $I^{(t)}$ is obtained by projecting the surface using $\Pi$. We
add colors to the input of the parametrization $\mathcal{P}$ and preserve them
after each transformation.

###  3.2 Classical approach

Let $\mathcal{M}_{\mathcal{S}}:=(\mathcal{V},\mathcal{E},\mathcal{F})$ be a
triangle mesh that approximates the smooth surface $\mathcal{S}$ with $n$
vertices $\mathcal{V}:=\\{\mathbf{v}_{i}\\}_{i\in[n_{v}]}$, edges
$\mathcal{E}\subseteq\mathcal{V}^{2}$ and triangular faces
$\mathcal{F}\subseteq\mathcal{V}^{3}$.

A typical strategy to formulate the SfT problem is to proceed in two steps.
First, in the registration step, we find correspondences between a known shape
and an image showing an unknown deformation with a feature-matching algorithm
providing

$\displaystyle
M^{(t)}:=\\{(\mathbf{s},\mathbf{i}):\mathbf{s}\in\mathcal{S}^{(t)},\mathbf{i}=\Pi(\mathbf{s})\in
I^{(t)}\\}\,.$ (2)

Then, in the reconstruction step, we obtain the deformed shape, which requires
inferring the depth of the image points obtained in registration.

Salzmann _et al_. [41] showed that the projection constraints given by the
registration algorithm can be expressed in a linear system of equations of the
form

$\displaystyle\mathbf{M}^{(t)}\mathbf{x}^{(t)}=\mathbf{0}\,,$ (3)

where $\mathbf{M}^{(t)}\in\mathbb{R}^{2|M^{(t)}|\times 3n_{v}}$ expresses the
correspondences and
$\mathbf{x}^{(t)}:=\text{vec}\left(\begin{bmatrix}\mathbf{v}_{1}^{(t)}&\cdots&\mathbf{v}_{N}^{(t)}\end{bmatrix}\right)$.

To compute $\mathbf{M}^{(t)}$, the tracked points on the surface are expressed
using barycentric coordinates. Recall that, given a point $\mathbf{p}$
belonging to facet $f$ of $\mathcal{M}_{\mathbf{S}}$, we have that
$\mathbf{p}=\sum_{i\in[3]}b_{i}\mathbf{v}_{f,i}$, where
$\sum_{i\in[3]}b_{i}=1$ and $\\{b_{i}\\}_{i\in[3]}$ are the barycentric
coordinates of $\mathbf{p}$.

Every solution to Eq. 3 re-projects correctly to the image, but the vertex
positions are not guaranteed to correspond with the true ones because of the
depth ambiguity. Given enough correspondences, the constraints given by the
registration step are enough and do not require knowing the depth [41].
However, the imperfection of image correspondences in real scenarios
introduces ambiguities, and the lack of identifiable texture on the surface
limits the number of possible matches among images. Overall, Eq. 3 is severely
under-constrained in practice, with around a third of the singular values of
$\mathbf{M}^{(t)}$ being very close to zero [41]. That means there are several
possible solutions, in this case, obtained by moving each point along the line
of sight. To factor out these incorrect solutions, we require additional
constraints.

Coltraro _et al_. [13] used the Riemannian metric preservation assumption to
constrain the possible vertices of a mesh for the simulation of deformable
surfaces. In particular, they introduced an easy-to-evaluate function $C$
that, given a parametrization $\varphi^{(t)}$ of
$\mathcal{M}_{\mathcal{S}}^{(t)}$, satisfies

$\displaystyle\mathbf{J}_{\varphi^{(t)}}^{T}\mathbf{J}_{\varphi^{(t)}}=\mathbf{J}_{\varphi^{\text{temp.}}}^{T}\mathbf{J}_{\varphi^{\text{temp.}}}\Longleftrightarrow
C(\mathbf{x}^{(t)})=\mathbf{0}\,,$ (4)

where $\varphi^{\text{temp.}}$ is the parametrization of the template shape.
Given the realistic simulations achieved by Coltraro _et al_. [13], we
consider the assumption to be reasonable.

As done in some works [31, 40], we can relax Eq. 3 and minimize the error in
the square sense subject to the metric preservation given by $C$. That is, for
$\Delta\mathbf{x}^{(t)}:=\mathbf{x}^{(t)}-\mathbf{x}^{(t-1)}$, the vertices of
the mesh can be found solving

$\displaystyle\begin{aligned}
\min_{\Delta\mathbf{x}^{(t)}}&\left\lVert\mathbf{M}^{(t)}(\mathbf{x}^{(t-1)}+\Delta\mathbf{x}^{(t)})\right\rVert^{2}_{2}\\\
\text{s.t.}&\mathbf{C}(\mathbf{x}^{(t-1)}+\Delta\mathbf{x}^{(t)})=\mathbf{0}\end{aligned}\,,$
(5)

which is a quadratic program with quadratic constraints. As done by Coltraro
_et al_. [13], to make the problem computationally tractable, we approximate
Eq. 5 with a sequence of quadratic programs with linear constraints using the
first order Taylor expansion
$\mathbf{C}(\mathbf{x}^{(t)})\approx\mathbf{C}(\mathbf{x}^{(t-1)})+\nabla\mathbf{C}(\mathbf{x}^{(t-1)})\Delta\mathbf{x}^{(t)}$
[13, 49].

Overall, for each time step $t$, the vertex positions of Classical are found
by solving the linear system

$\displaystyle\begin{cases}{\mathbf{M}^{(t)}}^{T}\mathbf{M}^{(t)}(\mathbf{x}^{(t-1)}+\Delta\mathbf{x}^{(t)})+\nabla\mathbf{C}(\mathbf{x}^{(t-1)})^{T}\boldsymbol{\lambda}=\mathbf{0}\\\
\mathbf{C}(\mathbf{x}^{(t-1)})+\nabla\mathbf{C}(\mathbf{x}^{(t-1)})\Delta\mathbf{x}^{(t)}=\mathbf{0}\end{cases}\,,$
(6)

where $\boldsymbol{\lambda}$ is the vector of Lagrange multipliers. We recall
that this loss function is not minimized during an offline training process
but instead during inference for each of the inputs.

###  3.3 Proposed method

Algorithm 1 Deformable Surface Reconstruction

1: Inputs: Template $\mathcal{T}$, matchings
$\\{M_{\mathcal{P}}^{(t)}\\}_{t\in[T]}$

2: Output: Estimated surfaces described by
$\\{\varphi_{\theta^{(t)}}\\}_{t\in[T]}$

3: Compute $P_{\mathcal{V}}$ according to $\mathcal{T}$

4:
$\displaystyle\theta^{\text{temp.}}\leftarrow\operatorname*{arg\,min}_{\theta^{\text{temp.}}}\frac{1}{|P_{\mathcal{V}}|}\sum_{\mathbf{p}_{i}\in
P_{\mathcal{V}}}\left\lVert\varphi_{\theta^{\text{temp.}}}(\mathbf{p}_{i})-\mathbf{v}_{i}\right\rVert_{2}$
$\rhd$Over-fit to template

5: Store
$\mathbf{J}_{\varphi_{\theta^{\text{temp.}}}}(\mathbf{p})^{T}\mathbf{J}_{\varphi_{\theta^{\text{temp.}}}}(\mathbf{p})$
$\forall\mathbf{p}\in P_{\mathcal{V}}$ $\rhd$Metric tensors of the template

6: Let $\varphi_{\theta^{(0)}}\leftarrow\varphi_{\theta^{\text{temp.}}}$
$\rhd$Needed for Eq. 12

7: for $t\in[T]$ do

8:
$\displaystyle\theta^{(t)}\leftarrow\operatorname*{arg\,min}_{\theta^{(t)}}\mathcal{L}_{\text{total}}$
$\rhd$Compute loss Eq. 13 with stored metric tensors

9: end for

Representing the state of a deformable object alone is an open challenge [53,
29]. Most SfT methods represent objects using meshes, but an alternative is to
use the explicit neural representation introduced by Groueix _et al_. [18].
Such a representation encodes a surface in the weights of a neural network
representing a mapping from 2D to 3D. Parametric representations are a general
way to express a surface, and their great flexibility allows to control
deformations with intuitive parameters [3]. Given a parametric surface, we can
analytically compute the first and second fundamental forms, Gaussian
curvature, and surface normals [8].

Implicit neural representations are another method that has recently emerged
as a promising alternative to classical discretized representations of signals
[55]. Both explicit and implicit neural representations can generate
continuous surfaces, which amounts to having meshes with an arbitrary
resolution. However, an advantage of using explicit representations in front
of the popular implicit representations such as NeRFS [28] or SDFs [35] is
that the co-domain of an explicit representation is the surface itself.
Therefore, if we want to generate a mesh, it is enough to sample the domain at
desired locations of the vertices. Instead, implicit representations require
using marching cubes on the outputs obtained by sampling many more points on
$\mathbb{R}^{3}$ in the case of SDFs (and the sampled points will only lie on
the surface if the output is 0), and sampling rays for NeRFs (with rays that
may or may not hit the object).

Given that both considered datasets only consist of rectangular surfaces, we
choose the parametrization $\mathcal{P}$ to be $\mathcal{P}:=[0,1]$ for
practical purposes. Other approaches [8, 18] set $\mathcal{P}$ as the interior
of the unit square. However, to naturally represent the surface edges, we
consider the closure of such a domain. The choice of $\mathcal{P}$ to be the
unit square allows representing all developable surfaces, _i.e_., smooth
surfaces that we can flatten into a plane, _e.g_., cylinders, cones, and
toruses.

To model non-developable surfaces like the sphere, we can choose the interior
of the unit square. In case of having surfaces with other topologies or with
holes, we could obtain $\mathcal{P}$ by conformal flattening of $\mathcal{T}$
[5], using a texture map [4], inferring the parametrization domain from data
[24], or combining different parametric surfaces to create an atlas [18].

A usual assumption in the SfT problem is that the template $\mathcal{T}$ and
the tracked surface at time $t$ $\mathcal{S}^{(t)}$ have the same topology,
which implies that they also share a parametrization space [5]. Therefore, we
can use the same choice of $\mathcal{P}$ to infer the surface at any point in
the sequence.

######  Proposition 1.

Let $\mathcal{S}$ be a surface that can be parametrized on the unit square.
There exists a feed-forward neural network with Softplus non-linearities that
can approximate $\mathcal{S}$ with arbitrary precision.

###### Proof.

The proof follows the same reasoning as in Groueix _et al_. [18, Proposition
2.], which states the same for ReLU non-linearities. Therefore, instead of
relying on the universal representation theorem by Hornik [19], this
proposition requires invoking the theorem by Kidger _et al_. [21], which works
with other activation functions, including the Softplus. ∎

Assuming that we can parametrize the surface of interest with the chosen
$\mathcal{P}$, we can leverage Proposition 1 and represent it with a feed-
forward neural network $\varphi_{\theta}$, where $\theta$ are the learnable
parameters. Note that the parametrization needs to be twice differentiable so
that the loss may incorporate first-order derivatives [8]. The requirement
motivates the use of Softplus [15], an approximation of the ReLU function with
smooth first and second derivatives [8].

Similarly to Brunet _et al_. [10], we obtain the surface parameters by
minimizing a combination of a re-projection loss Eq. 8, a term involving the
metric tensor favoring plausible poses Eq. 11, and a term that encourages
smooth motions Eq. 12. The key differences with this work are:

  * •

Re-projection loss: For a matching $(\mathbf{s},\mathbf{i})\in M^{(t)}$,
Brunet _et al_. [10] enforce that $\mathbf{s}\approx\Pi^{-1}(\mathbf{i})$.
This requires knowing the depth, which is included as an optimized parameter.
Instead, we check that $\Pi(\mathbf{s})\approx\mathbf{i}$.

  * •

Metric preservation loss: Our method enforces that the metric of the surface
at any time is close to that of the template. In contrast, Brunet _et al_.
[10] sets the metric tensor to the identity, which only allows modeling unit
squares on $\mathbb{R}^{3}$ when $\mathcal{P}=[0,1]$.

  * •

Smoothing loss: Brunet _et al_. [10] favor non-bending surfaces by minimizing
the Frobenius norm of the Hessian of the parametrization. Instead, we consider
the difference of surfaces in consecutive frames, which aims at reducing
frame-to-frame flickering.

For the sake of notation, let $\varphi_{\star}$ be the real parametrization of
$\mathcal{S}$. Let $P_{\mathcal{V}}$ be the set of points from $\mathcal{P}$
corresponding to the vertices of the template mesh and the surface meshes for
each time. That is

$\displaystyle
P_{\mathcal{V}}:=\\{\mathbf{p}\in\mathcal{P}:\mathbf{p}=\varphi_{\star}^{-1}(\mathbf{v})\
\forall\mathbf{v}\in\mathcal{V}\\}\,.$ (7)

Similarly to the relaxation of Eq. 2 used in Section 3.2, re-projection
consistency can be enforced with

$\displaystyle\mathcal{L}_{\text{projection}}:=\frac{1}{|M^{(t)}_{\mathcal{P}}|}\sum_{(\mathbf{p},\mathbf{i})\in
M^{(t)}_{\mathcal{P}}}\left\lVert\Pi(\varphi_{\theta^{(t)}}(\mathbf{p}))-\mathbf{i}\right\rVert_{2}\,,$
(8)

where

$\displaystyle
M_{\mathcal{P}}^{(t)}:=\\{(\mathbf{p},\mathbf{i}):\mathbf{p}\in\mathcal{P},\mathbf{i}=\Pi(\varphi_{\star}^{(t)}(\mathbf{p}))\in
I^{(t)}\\}\,.$ (9)

This term enforces that the recovered surfaces are consistent with their
corresponding images. This loss is present in all the SfT solutions, either
explicitly or implicitly in some neural network-based approaches, since the
monocular images provide the only information to recover the current state of
the surfaces.

Suppose the matches provide image locations for each vertex. One could
displace each 3D vertex location along the line of sight, thus obtaining
infinitely many surfaces that attain a zero re-projection loss. Enforcing
isometry ideally reduces the set of plausible solutions to a single surface
[4, Theorem 1].

To favor surfaces whose Riemannian metric is preserved, we add the loss term

$\displaystyle\mathcal{L}_{\text{metric}}:=\frac{1}{|P_{\mathcal{V}}|}\sum_{\mathbf{p}\in
P_{\mathcal{V}}}\left\lVert\right.$
$\displaystyle\mathbf{J}_{\varphi_{\theta^{(t)}}}(\mathbf{p})^{T}\mathbf{J}_{\varphi_{\theta^{(t)}}}(\mathbf{p})-$
(10)
$\displaystyle\left.\mathbf{J}_{\varphi_{\theta^{\text{temp.}}}}(\mathbf{p})^{T}\mathbf{J}_{\varphi_{\theta^{\text{temp.}}}}(\mathbf{p})\right\rVert_{F}^{2}\,,$
(11)

where $\left\lVert\cdot\right\rVert_{F}$ is the Frobenius norm.

Note that unlike works approximating the surface metric, we can compute it
analytically using surface parametrization [8]. Moreover, the preservation of
the metric is assessed in $P_{\mathcal{V}}$, not only in the visible parts,
which potentially helps to recover occluded zones [10].

Another difference with previous works is that we incorporate isometry as a
soft constraint, which differs from works imposing the metric to be exactly
preserved [4, 5], which may be a restrictive assumption in real scenarios.
Quasi-isometry, on the other hand, is a relatively mild constraint when
manipulating surfaces like clothes, as those are nearly inextensible [39].
This approximation is especially suited for robotics contexts, where ﬁne
details such as wrinkles are not needed [13].

Finally, the loss also includes a temporal regularization term

$\displaystyle\mathcal{L}_{\text{time}}:=\frac{1}{|P_{\mathcal{V}}|}\sum_{\mathbf{p}\in
P_{\mathcal{V}}}\left\lVert\varphi_{\theta^{(t)}}(\mathbf{p})-\varphi_{\theta^{(t-1)}}(\mathbf{p})\right\rVert_{2}\,.$
(12)

Adding a small amount of temporal regularization reduces frame-to-frame
flickering [54]. Additionally, point correspondences and metric preservation
are not sufficient to uniquely recover the correct surface [10]. In
particular, surface corners can freely bend as long as they do not shrink or
extend if there are no point correspondences [10].

The total loss used to update the parameters $\theta^{(t)}$ then becomes

$\displaystyle\mathcal{L}_{\text{total}}:=\mathcal{L}_{\text{projection}}+\lambda_{\text{metric}}\mathcal{L}_{\text{metric}}+\lambda_{\text{time}}\mathcal{L}_{\text{time}}\,.$
(13)

In Algorithm 1, we detail the procedure on how to estimate the surfaces for
the whole sequence given the inputs of the SfT problem.

##  4 Experiments

Dataset $\downarrow$ / Methods$\rightarrow$ | GP | Lap | Dense | Graph | Texless | Classical | Ours
---|---|---|---|---|---|---|---
| Brick | 109.45 | 44.80 | 81.56 | 87.03 | 84.99$\dagger$ | 67.25 | 48.32
| Campus | 85.48 | 85.80 | 57.00 | 76.66 | 155.16 | 66.00 | 40.18
| Cloth | 104.30 | 460.44 | 85.58 | 95.08 | 150.53 | 891.65 | 54.49
| Cobble | 100.54 | 41.90 | 68.90 | 66.23 | 88.94 | 56.33 | 53.05
| Cushion I | 104.17 | 72.55 | 108.65 | 124.23 | 100.22$\dagger$ | 252.36 | 88.97
| Cushion II | 103.95 | 157.76 | 96.91 | 147.95 | 226.35$\ddagger$ | 918.70 | 71.23
| Newspaper I | 83.25 | 63.23 | 85.12 | 97.55 | 130.26$\dagger$ | 79.61 | 43.23
| Newspaper II | 79.88 | 67.66 | 73.65 | 87.33 | 186.14 | 68.39 | 46.37
| Scene | 102.25 | 57.36 | 82.37 | 70.74 | 136.24 | 69.85 | 60.14
| Stone | 100.62 | 421.36 | 90.93 | 91.28 | 126.62$\dagger$ | 1140.05 | 72.03
DeSurT | Sunset | 107.89 | 248.29$\ast$ | 67.24 | 84.43 | 138.52$\ddagger$ | 72.97 | 58.77
| Lr_bottom_edge | 0.10 | 0.88$\ast$ | 0.09 | 1.09 | 0.09$\dagger$ | 0.07 | 0.06
| Lr_bottom_edge_tl_corn | 0.06 | 3.00$\ast$ | 0.07 | 1.08 | 0.06$\dagger$ | 0.04 | 0.05
| Lr_left_edge | 0.08 | 1.60$\ast$ | 0.10 | 1.05 | 0.08$\dagger$ | 0.06 | 0.05
| Lr_tl_tr_corns | 0.07 | 0.85$\ast$ | 0.09 | 1.00 | 0.06$\dagger$ | 0.06 | 0.05
| Lr_top_edge_1 | 0.09 | 0.51$\ast$ | 0.09 | 0.96 |  0.08$\dagger$ | 0.07 | 0.07
| Lr_top_edge_2 | 0.08 | 0.60$\ast$ | 0.09 | 0.97 |  0.07$\dagger$ | 0.06 | 0.07
TDS | Lr_top_edge_3 | 0.06 | 0.36$\ast$ | 0.07 | 1.02 | 0.07$\ddagger$ | 0.05 | 0.06

Table 2: Quantitative results. This table shows the mean tracking error (mm)
obtained when reconstructing a mesh from monocular images. The best value for
each sequence (the lowest) is shown in boldface and the runner-up is
underlined. Asterisks ($\ast$) indicate that a method did not terminate, in
which case the average error of the meshes obtained before the method crashed
is reported. The sequences used to train and validate Texless are indicated
with a dagger ($\dagger$) and a double dagger ($\ddagger$), respectively. Note
that we favor the algorithm by providing it with full sequences and in some
cases reporting the values on training examples.

Methods |  Time (s) | DeSurT | TDS
---|---|---|---
| GP | $\mathbf{0.01}$ | $98.34\pm 10.37$ | $0.07\pm\mathbf{0.01}$
Learning-based | Texless | $\mathbf{0.01}$ | $138.54\pm 41.72$ | $0.07\pm\mathbf{0.01}$
| Lap | $5.80$ | $156.47\pm 153.31$ | $1.11\pm 0.92$
| Dense | $26.73$ | $81.63\pm\mathbf{14.55}$ | $0.09\pm\mathbf{0.01}$
| Graph | $32.52$ | $93.50\pm 23.75$ | $1.02\pm 0.05$
| Classical | $49.00$ | $334.83\pm 424.56$ | $\mathbf{0.06\pm 0.01}$
Optimization-based | Ours | $\mathbf{0.95}$ | $\mathbf{57.80}\pm 14.72$ | $\mathbf{0.06\pm 0.01}$

Table 3: Additional quantitative performance indicators. The column Time shows
the average the number of seconds required to process one image for the TDS
sequence Lr_bottom_edge, the dataset with higher resolution meshes
(concretely, 961 vertices). The columns DeSurT and TDS present the mean and
standard deviation of the tracking error in millimeters across the sequences
of each dataset. The best value for each sequence (the lowest for all metrics)
is shown in boldface.

###  4.1 Datasets

To quantitatively evaluate the proposed method, we require ground-truth mesh
vertex positions. In this paper, we use two public datasets involving
rectangular deformable surfaces:

  * •

DeSurT [52]: Dataset consisting of 11 video streams, with around 300 images
each, displaying different types of materials, deformations, and lighting
conditions. The surfaces are either well-textured (Campus, Cobble, Cushion I,
Scene, Newspaper I, and Newspaper II), repetitively textured (Brick, Cloth,
and Cushion II), or weakly textured (Stone and Sunset). The DeSurT dataset
represents surfaces with meshes of size $13\times 10$ vertices for all the
sequences but the cushion, which uses an $11\times 11$ mesh.

  * •

TDS [7]: Dataset showing deformable surfaces under various lighting
conditions. We use all the sequences with ground-truth vertex annotations,
which account for seven image sequences with around 900 images each,
displaying a piece of cloth represented with a $31\times 31$ mesh.

(a) Ground truth

(b) GP

(c) Lap

(d) Dense

(e) Graph

(f) Texless

(g) Classical

(h) Ours

Figure 3: Qualitative results. Comparison of the reconstructions obtained by
different methods on the frame 85 of the campus sequence of DeSurT [52].

###  4.2 Baselines

We use the following methods to compare the performance of the proposed
technique. Unless specified, we use the publicly available code implemented by
the original authors without modifying the algorithms or the hyper-parameters.

  * •

GP: Unconstrained GP-LVM [49]. We exclude the constrained GP-LVM introduced in
the same paper from the comparisons as it consistently underperformed the
unconstrained version.

  * •

Lap: Method based on Laplacian meshes [27, 31, 56].

  * •

Dense: Dense image registration algorithm by Ngo _et al_. [30].

  * •

Graph: Deformable surface tracking using graph matching [52].

  * •

TexLess: The model from Bednarík _et al_. [7] trained from scratch with
ground-truth mesh vertices for each dataset and mesh resolution.

  * •

Classical: Classical method outlined in Section 3.2.

  * •

Ours: Proposed method described in Section 3.3.

###  4.3 Implementation details

Classical requires solving a very sparse linear system, _i.e_., Eq. 6,
multiple times to approximate Eq. 5. In practice, such a linear system is
solved in the least-squares sense using the SciPy [50] routine limited to 100
iterations multiple times until the approximation is good enough as understood
by the same criterion of Coltraro _et al_. [13].

All the surfaces of the datasets considered in this work are rectangular and
represented using a mesh with equispaced vertices. For this reason, we can
define $P_{\mathcal{V}}$ with the coordinates given by a grid on the unit
square using the width and height of the mesh. Concretely, we follow the
convention to assign those values using the ordering of the vertices as seen
on the template image (see Fig. 2 for an illustration of the parametrization
on the $RG$ color space).

The matches $M^{(t)}$ can be obtained with several methods such as SIFT [25],
SURF [6], Ferns [32], or soft graph matching [52] and then applying an outlier
rejection mechanism [51]. Obtaining the matches with a dedicated model and
then performing reconstruction results in correspondences robust to
occlusions, especially for well-textured surfaces [30].

We obtain correspondences with the registration algorithm used in the baseline
Lap, based on matching SIFT [25] features and then applying an outlier
rejection mechanism. The Lap algorithm did not terminate and hence could not
provide matchings for all the frames indicated with an asterisk ($\ast$) in
Table 2. In this case, we used synthetic matches for simplicity by sampling a
random point inside each facet of the mesh. To set the weight for each
regularization term in (13), we perform a grid search on the evaluation loss
testing the values

$\displaystyle\lambda_{\text{metric}},\lambda_{\text{time}}\in\\{0,10^{-2},10^{-1},\dots,10^{2}\\}$
(14)

on the first sequence for each dataset. We use
$(\lambda_{\text{metric}},\lambda_{\text{time}})=(0.01,0.001)$ on DeSurT and
$(\lambda_{\text{metric}},\lambda_{\text{time}})=(100,100)$ on TDS. We can
justify the high regularization values on TDS because the TDS sequences show a
cloth pinned to a fixed bar along a given edge or corners [7], so the position
of at least part of the cloth was quite stable, which is exploited by both the
temporal and metric constraints. Finally, although we used synthetic matches
in this case, the best results obtained by our method were achieved when the
relative contribution of the projection loss was the lowest.

We represent the surface parametrization with a multi-layer perceptron with
three hidden layers having 128, 256, and 128 units and implemented using the
PyTorch [36] framework. We obtain the parameters of this model by minimizing
Eq. 13 using the ADAM optimizer [22].

###  4.4 Evaluation

To evaluate the accuracy of the proposed model, we compute the Euclidean
distances from vertex to vertex. In particular, Table 2 reports the mean of
such distances, _i.e_.,

$\displaystyle\frac{1}{T}\sum_{t\in[T]}\left[\frac{1}{N}\sum_{n\in[N]}\left\lVert\hat{\mathbf{v}}_{n}^{(t)}-\mathbf{v}_{n}^{(t)}\right\rVert_{2}\right]\,,$
(15)

the quantity reported in several 3D reconstruction works [30, 37, 49, 52].
Given a parametrization $\varphi_{\theta^{(t)}}$ obtained with the proposed
method, one can compute $\hat{\mathbf{v}}_{n}^{(t)}$ by evaluating the
parametrization at the point in $P_{\mathcal{V}}$ corresponding to the $n-$th
vertex.

It is worth noting that the best performance for all sequences is attained by
methods relying on feature matches. In particular, Lap achieves the best
performance for some sequences but fails on others, which is consistent with
the results in Kairanda _et al_. [20]. The superiority of algorithms taking
matches as input contrasts with the current trend of directly predicting
surfaces directly from images. However, relying on an external registration
algorithm is a double-edged sword and becomes the main limitation of the
proposed method. The reason is that matching algorithms introduce noise and
fail with repetitive or poorly textured surfaces (_e.g_., TDS dataset).

An alternative is to use dense approaches like Dense, which does not extract
features but instead maximizes a similarity measure to perform registration
[30, 54]. These approaches usually need consistent illumination and suffer
from brightness changes, occlusions, and motion blur [52].

Data-based approaches like Texless do not require matches but typically work
only with the surface seen during training and require fine-tuning to
different templates. Fuentes-Jimenez _et al_. [16] attempted training texture-
generic neural networks, but their results are still less accurate than the
ones obtained with texture-specific methods. Table 2 showcases that data-based
approaches did not perform strictly better than other methods for any tested
sequences.

Table 3 reports the average time required by each method to process one image
and statistics about the performance across all the sequences reported in
Table 2. On the one hand, as expected, learning-based methods attain the
lowest inference time as they only need to evaluate a function. Optimization-
based methods require finding the best surface parameters given an image,
which requires performing several function evaluations and parameter updates.
The proposed method is optimization-based but has significantly lower
computational overhead than its competitors. On the other hand, the proposed
approach attains the best average performance across both tested datasets.
Moreover, the standard deviation of mean tracking errors for different
sequences is among the lowest, which shows that our method is generally
applicable and robust.

In Fig. 3, we show a qualitative evaluation of the obtained results. We depict
the projection of the reconstructed mesh on top of the input image for each
tested method and the ground truth. Note that, as mentioned above, a perfect
vertex projection does not guarantee that the reconstruction is correct due to
the depth ambiguity. Therefore, one must consider both qualitative and
quantitative results, the former considering how well the projected mesh
matches the image and the latter considering the estimation error in 3D.

While Classical achieves one of the lowest mean tracking errors, some
recovered surfaces are irregular. Dense attains the second-best reconstruction
performance on the campus sequence despite failing to recover the surface in
some individual frames. This approach does not rely on an external feature-
matching algorithm and instead uses dense matching of image features in the
optimized cost function. Although this is beneficial for poorly textured
surfaces, it may lead to suboptimal re-projection constraints for identifiable
textures. However, the quantitative metric being one of the best, shows that
it better resolves the depth ambiguities than other methods.

The Lap and Graph methods also yield one of the best quantitative results on
the campus sequence, this time reflected in qualitatively faithful surface
reconstructions. Nonetheless, we can see mismatches with the surface contour
w.r.t. to the Ground truth. Ours attains the best quantitative performance in
Table 2, shows good qualitative results, and is better at recovering the
surface contours than the alternatives.

##  5 Conclusions

This work tackles the inherently ill-posed problem of reconstructing
deformable surfaces from monocular images. The proposed method assumes that
the Riemannian metric of the manipulated surface is approximately constant or
equivalently that the transformation from the template object is an isometry.
Metric preservation consists of a mild hypothesis for a wide variety of
surfaces since many materials do not perceptibly shrink or stretch when they
suffer deformations [38].

The Riemannian metric preservation constraint proposed by Coltraro _et al_.
[13] for cloth simulation can easily be incorporated into the SfT problem,
which led to the approach denoted as Classical. The results obtained with this
approach achieve one of the best performances for the TDS sequences (see Table
2). Despite the notable results, which back up the metric preservation
assumption, this approach scales poorly with the number of vertices and
diverges for some sequences (see _e.g_., the Stone sequence in Table 2). For
this reason, instead of naively incorporating the constraints and optimizing
the vertex positions, we propose to use explicit neural surfaces.

Parametric surfaces learned by neural networks pose an attractive framework to
represent surfaces with arbitrary precision, an observation formalized in
Proposition 1. Having a continuous surface, we avoid discretization problems
when estimating the surface parameters and can generate meshes with different
levels of detail. The learned surface parametrization allows for the
analytical computation of differential geometric quantities. In this work, we
used the well-known isometry constraint, but we could easily modify the
proposed framework to enforce other constraints, such as constant surface area
as done by Salzmann _et al_. [40].

Another advantage of using neural networks is that we can apply non-convex
constraints involving partial derivatives and, therefore, not rely on the
relaxations proposed by Coltraro _et al_. [13] needed in a classical
optimization framework. Contrasting to the previous methods for SfT using
neural networks, our approach does not require offline training. Therefore, it
lifts the requirement of a dataset having enough samples to represent the
dynamics and appearance of an object, thus overcoming the problems of the
data-based approaches described in Section 2.2. Among others, the consequences
are that we can apply the proposed method to any sequence without modification
and that it does not require a dataset to infer the manifold of plausible
deformations from data. The solution to avoid training is to use an iterative
optimization process for each input, but such optimization takes orders of
magnitude less time than the alternative methods considered in this work.

## Acknowledgments

This work is part of the project CLOTHILDE (“CLOTH manIpulation Learning from
DEmonstrations”) which has received funding from the European Research Council
(ERC) under the European Union’s Horizon 2020 research and innovation program
(Advanced Grant agreement No. 741930). O.B. thanks Franco Coltraro and Xavier
Gràcia for their insights on differential geometry, and the European
Laboratory for Learning and Intelligent Systems (ELLIS) PhD program for
support.

## References

  * [1] Ferran Alet, Dylan Doblar, Allan Zhou, Joshua B. Tenenbaum, Kenji Kawaguchi, and Chelsea Finn.  Noether Networks: Meta-Learning Useful Conserved Quantities.  In NeurIPS, pages 16384–16397, 2021.
  * [2] Ferran Alet, Kenji Kawaguchi, Tomás Lozano-Pérez, and Leslie Pack Kaelbling.  Tailoring: Encoding inductive biases by optimizing unsupervised objectives at prediction time.  In NeurIPS, pages 29206–29217, 2021.
  * [3] Veronica E. Arriola-Rios, Puren Guler, Fanny Ficuciello, Danica Kragic, Bruno Siciliano, and Jeremy L. Wyatt.  Modeling of Deformable Objects for Robotic Manipulation: A Tutorial and Review.  Frontiers in Robotics and AI, 7(82):1–25, 2020.
  * [4] A. Bartoli, Y. Gérard, F. Chadebecq, and T. Collins.  On template-based reconstruction from a single view: Analytical solutions and proofs of well-posedness for developable, isometric and conformal surfaces.  In CVPR, pages 2026–2033, 2012.
  * [5] Adrien Bartoli, Yan Gérard, François Chadebecq, Toby Collins, and Daniel Pizarro.  Shape-from-Template.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(10):2099–2118, 2015.
  * [6] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool.  SURF: Speeded Up Robust Features.  In ECCV, pages 404–417, 2006.
  * [7] Jan Bednarík, Pascal V. Fua, and Mathieu Salzmann.  Learning to reconstruct texture-less deformable surfaces from a single view.  In 3DV, pages 606–615, 2018.
  * [8] Jan Bednarík, Shaifali Parashar, Erhan Gundogdu, Mathieu Salzmann, and P. Fua.  Shape reconstruction by learning differentiable surface representations.  In CVPR, pages 4716–4725, 2020.
  * [9] Leon Bodenhagen, Andreas R. Fugl, Andreas Jordt, Morten Willatzen, Knud A. Andersen, Martin M. Olsen, Reinhard Koch, Henrik G. Petersen, and Norbert Krüger.  An Adaptable Robot Vision System Performing Manipulation Actions With Flexible Objects.  IEEE Transactions on Automation Science and Engineering, 11(3):749–765, 2014.
  * [10] Florent Brunet, Richard Hartley, Adrien Bartoli, Nassir Navab, and Rémy Malgouyres.  Monocular Template-Based Reconstruction of Smooth and Inextensible Surfaces.  In ACCV, pages 52–66, 2010.
  * [11] David Casillas-Pérez, Daniel Pizarro, David Fuentes-Jimenez, Manuel Mazo, and Adrien Bartoli.  The Isowarp: The Template-Based Visual Geometry of Isometric Surfaces.  IJCV, 129(7):2194–2222, 2021.
  * [12] Ajad Chhatkuli, Daniel Pizarro, Adrien Bartoli, and Toby Collins.  A Stable Analytical Framework for Isometric Shape-from-Template by Surface Integration.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(5):833–850, 2017.
  * [13] Franco Coltraro, Jaume Amorós, Maria Alberich-Carramiñana, and Carme Torras.  An inextensible model for the robotic manipulation of textiles.  Applied Mathematical Modelling, 101:832–858, 2022.
  * [14] Manfredo P Do Carmo.  Differential geometry of curves and surfaces: revised and updated second edition.  Courier Dover Publications, 2016.
  * [15] Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia.  Incorporating Second-Order Functional Knowledge for Better Option Pricing.  In NeurIPS, pages 472–478, 2000.
  * [16] David Fuentes-Jimenez, Daniel Pizarro, David Casillas-Pérez, Toby Collins, and Adrien Bartoli.  Texture-Generic Deep Shape-From-Template.  IEEE Access, 9:75211–75230, 2021.
  * [17] David Fuentes-Jimenez, Daniel Pizarro, David Casillas-Pérez, Toby Collins, and Adrien Bartoli.  Deep Shape-from-Template: Single-image quasi-isometric deformable registration and reconstruction.  Image and Vision Computing, 127(104531):1–19, 2022.
  * [18] Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan Russell, and Mathieu Aubry.  AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation.  In CVPR, pages 216–224, 2018.
  * [19] Kurt Hornik.  Approximation capabilities of multilayer feedforward networks.  Neural Networks, 4(2):251–257, 1991.
  * [20] Navami Kairanda, Edith Tretschk, Mohamed Elgharib, Christian Theobalt, and Vladislav Golyanik.  $\varphi$-SfT: Shape-from-Template with a Physics-Based Deformation Model.  In CVPR, pages 3938–3948, 2022.
  * [21] Patrick Kidger and Terry J. Lyons.  Universal Approximation with Deep Narrow Networks.  In COLT, pages 2306–2327, 2020.
  * [22] Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  In ICLR, pages 1–15, 2015.
  * [23] Jose Lamarca, Shaifali Parashar, Adrien Bartoli, and J. M. M. Montiel.  DefSLAM: Tracking and Mapping of Deforming Scenes From Monocular Sequences.  IEEE Transactions on Robotics, 37(1):291–303, 2021.
  * [24] Jiahui Lei, Srinath Sridhar, Paul Guerrero, Minhyuk Sung, Niloy Mitra, and Leonidas J. Guibas.  Pix2Surf: Learning Parametric 3D Surface Models of Objects from Images.  In ECCV, pages 121–138, 2020.
  * [25] D.G. Lowe.  Object recognition from local scale-invariant features.  In ICCV, pages 1150–1157, 1999.
  * [26] Adrià Luque, David Parent, Adrià Colomé, Carlos Ocampo-Martinez, and Carme Torras.  Model Predictive Control for Dynamic Cloth Manipulation: Parameter Learning and Experimental Validation.  arXiv, 2022.
  * [27] Stéphane Magnenat, Dat Tien Ngo, Fabio Zünd, Mattia Ryffel, Gioacchino Noris, Gerhard Rothlin, Alessia Marra, Maurizio Nitti, Pascal Fua, Markus Gross, and Robert W. Sumner.  Live texturing of augmented reality characters from colored drawings.  IEEE Transactions on Visualization and Computer Graphics, 21(11):1201–1210, 2015.
  * [28] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.  In ECCV, pages 405–421, 2020.
  * [29] J. Montagnat, H. Delingette, and N. Ayache.  A review of deformable surfaces: topology, geometry and deformation.  Image and Vision Computing, 19(14):1023–1040, 2001.
  * [30] Dat Tien Ngo, Sanghyuk Park, Anne Jorstad, Alberto Crivellaro, Chang D. Yoo, and Pascal Fua.  Dense Image Registration and Deformable Surface Reconstruction in Presence of Occlusions and Minimal Texture.  In ICCV, pages 2273–2281, 2015.
  * [31] Tien Dat Ngo, Jonas Östlund, and Pascal Fua.  Template-based Monocular 3D Shape Recovery using Laplacian Meshes.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(1):172–187, 2016.
  * [32] Mustafa Ozuysal, Michael Calonder, Vincent Lepetit, and Pascal Fua.  Fast Keypoint Recognition Using Random Ferns.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(3):448–461, 2010.
  * [33] Shaifali Parashar and Adrien Bartoli.  3DVFX: 3D Video Editing using Non-Rigid Structure-from-Motion.  In Paolo Cignoni and Eder Miguel, editors, Eurographics - Short Papers, pages 29–32, 2019.
  * [34] Shaifali Parashar, Daniel Pizarro, Adrien Bartoli, and Toby Collins.  As-Rigid-As-Possible Volumetric Shape-From-Template.  In ICCV, pages 891–899, December 2015.
  * [35] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation.  In CVPR, pages 165–174, 2019.
  * [36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In NeurIPS, pages 8026–8037, 2019.
  * [37] A. Pumarola, A. Agudo, L. Porzi, A. Sanfeliu, V. Lepetit, and F. Moreno-Noguer.  Geometry-Aware Network for Non-Rigid Shape Prediction from a Single View.  In CVPR, pages 4681–4690, 2018.
  * [38] Mathieu Salzmann and Pascal Fua.  Reconstructing sharply folding surfaces: A convex formulation.  In CVPR, pages 1054–1061, 2009.
  * [39] Mathieu Salzmann and Pascal Fua.  Linear Local Models for Monocular Reconstruction of Deformable Surfaces.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5):931–944, 2011.
  * [40] Mathieu Salzmann, Richard Hartley, and Pascal Fua.  Convex Optimization for Deformable Surface 3-D Tracking.  In ICCV, pages 1–8, 2007.
  * [41] Mathieu Salzmann, Vincent Lepetit, and Pascal Fua.  Deformable Surface Tracking Ambiguities.  In CVPR, pages 1–8, 2007.
  * [42] Mathieu Salzmann, Francesc Moreno-Noguer, Vincent Lepetit, and Pascal Fua.  Closed-Form Solution to Non-rigid 3D Surface Registration.  In CVPR, pages 581–594, 2008.
  * [43] Mathieu Salzmann, Julien Pilet, Slobodan Ilic, and Pascal Fua.  Surface Deformation Models for Nonrigid 3D Shape Recovery.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(8):1481–1487, 2007.
  * [44] Mathieu Salzmann and Raquel Urtasun.  Implicitly Constrained Gaussian Process Regression for Monocular Non-Rigid Pose Estimation.  In NeurIPS, pages 2065–2073, 2010.
  * [45] Mathieu Salzmann, Raquel Urtasun, and Pascal Fua.  Local deformation models for monocular 3D shape recovery.  In CVPR, pages 1–8, 2008.
  * [46] Jose Sanchez, Juan-Antonio Corrales, Belhassen-Chedli Bouzgarrou, and Youcef Mezouar.  Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey.  The International Journal of Robotics Research, 37(7):688–716, 2018\.
  * [47] Olga Sorkine and Marc Alexa.  As-Rigid-As-Possible Surface Modeling.  In Geometry Processing, pages 109–116. The Eurographics Association, 2007.
  * [48] Edith Tretschk, Navami Kairanda, Mallikarjun B R, Rishabh Dabral, Adam Kortylewski, Bernhard Egger, Marc Habermann, Pascal Fua, Christian Theobalt, and Vladislav Golyanik.  State of the Art in Dense Monocular Non-Rigid 3D Reconstruction.  arXiv, 2022.
  * [49] Aydin Varol, Mathieu Salzmann, Pascal Fua, and Raquel Urtasun.  A Constrained Latent Variable Model.  In CVPR, pages 2248–2255, 2012.
  * [50] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  Nature Methods, 17(3):261–272, 2020.
  * [51] Christian Vogler, Siome Goldenstein, Jorge Stolfi, Vladimir Pavlovic, and Dimitris Metaxas.  Outlier rejection in high-dimensional deformable models.  Image and Vision Computing, 25(3):274–284, 2007.
  * [52] Tao Wang, Haibin Ling, Congyan Lang, Songhe Feng, and Xiaohui Hou.  Deformable Surface Tracking by Graph Matching.  In ICCV, pages 901–910, 2019.
  * [53] Hang Yin, Anastasia Varava, and Danica Kragic.  Modeling, learning, perception, and control methods for deformable object manipulation.  Science Robotics, 6(54):1–16, 2021.
  * [54] Rui Yu, Chris Russell, Neill D. F. Campbell, and Lourdes Agapito.  Direct, Dense, and Deformable: Template-Based Non-rigid 3D Reconstruction from RGB Video.  In ICCV, pages 918–926, 2015.
  * [55] Gizem Yüce, Guillermo Ortiz-Jiménez, Beril Besbinar, and Pascal Frossard.  A structured dictionary perspective on implicit neural representations.  In CVPR, pages 19228–19238, 2022.
  * [56] Jonas Östlund, Aydin Varol, Dat Ngo, and Pascal Fua.  Laplacian Meshes for Monocular 3D Shape Recovery.  In ECCV, pages 412–425, 2012.