# Continuous Surface Embeddings

Natalia Neverova David Novotny Vasil Khalidov Marc Szafraniec
Patrick Labatut Andrea Vedaldi
{nneverova, dnovotny, vkhalidov, mszafraniec, plabatut<EMAIL_ADDRESS>
Facebook AI Research

###### Abstract

In this work, we focus on the task of learning and _representing dense
correspondences_ in deformable object categories. While this problem has been
considered before, solutions so far have been rather ad-hoc for specific
object types (i.e., humans), often with significant manual work involved.
However, scaling the geometry understanding to all objects in nature requires
more automated approaches that can also express correspondences between
related, but geometrically different objects. To this end, we propose a new,
learnable image-based representation of dense correspondences. Our model
predicts, for each pixel in a 2D image, an embedding vector of the
corresponding vertex in the object mesh, therefore establishing dense
correspondences between image pixels and 3D object geometry. We demonstrate
that the proposed approach performs on par or better than the state-of-the-art
methods for dense pose estimation for humans, while being conceptually
simpler. We also collect a new in-the-wild dataset of dense correspondences
for animal classes and demonstrate that our framework scales naturally to the
new deformable object categories.

##  1 Introduction

Understanding the geometry of natural objects, such as humans and other
animals, must start from the notion of _correspondence_. Correspondences tell
us which parts of different objects are geometrically equivalent, and thus
form the basis on which an understanding of geometry can be developed. In this
paper, we are interested in particular in learning and computing
correspondence starting from 2D images of the objects, a preliminary step for
3D reconstruction and other applications.

While the correspondence problem has been considered many times before, most
solutions still involve a significant amount of manual work. Consider for
example a state-of-the-art method such as DensePose [17]. Given a new object
category to model with DensePose, one must start by defining a _canonical
shape_ $S$, a sort of ‘average’ 3D shape used as a reference to express
correspondences. Then, a dataset of images of the object must be collected and
annotated with millions of manual point correspondences between the images and
the canonical 3D model. Finally, the model must be manually partitioned into a
number of parts, or charts, and a deep neural network must be trained to
segment the image and regress the $uv$ coordinates for each chart, guided by
the manual annotations, yielding a DensePose predictor. Given a new category,
this process must be repeated from scratch.

Figure 1:  Overview. Compared to DensePose [17], our CSE framework is
conceptually simpler and is directly extendable to multi-class problems
through learning a universal dense pose network. (Color coding for each class
mesh in the figure is chosen arbitrarily and independently from others.)

There are some obvious scalability issues with this approach. The most
significant one is that the entire process must be repeated for each new
object category one wishes to model. This includes the laborious step of
collecting annotations for the new class. However, categories such as animals
share significant similarities between them; for instance, recently [44] has
shown that DensePose trained on humans transfers well on chimpanzees. Thus, a
much better scalable solution can be obtained by _sharing_ training data and
models between classes. This brings us to the second shortcoming of DensePose:
the nature of the model makes it difficult to realize this information
sharing. In particular, the need for breaking the canonical 3D models into
different charts makes relating different models cumbersome, particularly in a
learning setup.

One important contribution of this paper is to introduce a better and more
flexible representation of correspondences that can be used as a drop-in
replacement in architectures such as DensePose. The idea is to introduce a
_learnable positional embedding_. Namely, we associate each point $X$ of the
canonical model $S$ to a compact _embedding vector_ $e_{X}$, which provides a
deformation-invariant representation of the point identity. We also note that
the embedding can be interpreted as a smoothly-varying function defined over
the 3D model $S$, interpreted as a manifold. As such, this allows us to use
the machinery of _functional maps_ [39] to work with the embeddings, with two
important advantages: (1) being able to significantly reduce the
dimensionality of the representation and (2) being able to efficiently relate
representations between models of different object categories.

Empirically, we show that we can learn a deep neural network that predicts,
for each pixel in a 2D image, the embedding vector of the corresponding object
point, therefore establishing dense correspondences between the image pixels
and the object geometry. For humans, we show that the resulting
correspondences are as or more accurate than the reference state-of-the-art
DensePose implementation, while achieving a significant simplification of the
DensePose framework by removing the need of charting the model. As an
additional bonus, this removes the ‘seams’ between the parts that affect
DensePose. Then, we use the ability of the functional maps to relate different
3D shapes to help transferring information between different object
categories. With this, and a very small amount of manual training data, we
demonstrate for the first time that a _single (universal) DensePose network_
can be extended to capturing multiple animal classes with a high degree of
sharing in compute and statistics. The overview of our method with learning
_continuous surface embeddings_ (CSE) is shown in Fig. 1b (for comparison, the
DensePose [17] setup (IUV) is shown in Fig. 1a).

##  2 Related work

#### Human pose recognition.

With deep learning, image-based human pose estimation has made substantial
progress [52, 37, 12], also due to the availability of large datasets such as
COCO [31], MPII [3], Leeds Sports Pose Dataset (LSP) [23, 24], PennAction
[58], or PoseTrack [2]. Our work is most related with DensePose [17], which
introduced a method to establish dense correspondences between image pixels
and points on the surface of the average SMPL human mesh model [32].

#### Unsupervised pose recognition.

Most pose estimators [5, 49, 7, 50, 43, 48, 33, 59, 22] require full
supervision, which is expensive to collect, especially for a model such as
DensePose. A handful of works have tackled this issue by seeking unsupervised
and weakly-supervised objectives, using cues such as equivariance to synthetic
image transformations. The most relevant to us is Slim DensePose [36], which
showed that DensePose annotations can be significantly reduced without
incurring a large performance penalty, but did not address the issue of
scaling to multiple classes.

#### Animal pose recognition.

Compared to humans, animal pose estimation is significantly less explored.
Some works specialise on certain animals (tigers [30], cheetahs [35] or
drosophila melanogaster flies [19]). Tulsiani et al. [51] transfer pose
between annotated animals and un-annotated ones that are visually similar.
Several works have focused on animal landmark detection. Rashid et al. [41]
and Yang et al. [55] studied animal facial keypoints. A well explored class
are birds due to the CUB dataset [53]. Some works [57, 45] proposed various
types of detectors for birds, while others explored reconstructing sparse [38]
and dense [26, 25, 13] 3D shapes. Beyond birds, Zuffi et al. [61, 62, 60] have
explored systematically the problem of reconstructing 3D deformable animal
models from image data. They utilise the SMAL animal shape model, which is an
analogue for animals of the more popular human SMPL model for humans [32].
While [61, 62] are based on fitting 2D keypoints at test time, Biggs et al.
[6] directly regresses the 3D shape parameters instead. Recently, Kulkarni et
al. [28, 29] leveraged canonical maps to perform the 3D shape fitting as well
as establishing of dense correspondences across different instances of an
animal species.

#### 3D shape analysis.

Our work is also related to the literature that studies the intrinsic geometry
of 3D shapes. Early approaches [15, 9] analysed shapes by performing multi-
dimensional scaling of the geodesic distances over the shape surface, as these
are invariant to isometric deformations. Later, Coifman and Lafon [14]
popularized the diffusion geometry due to its increased robustness to small
perturbations of the shape topology. The seminal work of Rustamov [42]
proposed to use the eigenfunctions of the Laplace-Beltrami operator (LBO) on a
mesh to define a basis of functions that smoothly vary along the mesh surface.
The LBO basis was later leveraged in other diffusion descriptors such as the
heat kernel signature (HKS) [47], wave kernel signature (WKS) [4] or Gromov-
Hausdorff descriptors [10]. A scale-invariant version of HKS was introduced in
[11], while [8] proposed an HKS-based equivalent of the image BoW descriptor
[46]. While HKS/WKS establish ‘hard’ correspondences between individual points
on shapes, Ovsjanikov et al. [39] introduced the functional maps (FM) that
align shapes in a soft manner by finding a linear map between spaces of
functions on meshes. Interestingly, [39] has revealed an intriguing connection
between FMs and their efficient representation using the LBO basis. The FM
framework became popular and was later extended in [40, 27, 1]. Relevantly to
us, [34] proposed ZoomOut, a method that estimates FM in a multi-scale
fashion, which we improve for our species-to-species mesh correspondences.

##  3 Method

We propose a new approach for representing continuous correspondences (or
surface embeddings, CSE) between an image and points in a 3D object. To this
end, let $S\subset\mathbb{R}^{3}$ be a _canonical surface_. Each point $X\in
S$ should be thought of as a ‘landmark’, i.e. an object point that can be
identified consistently despite changes in viewpoint, object deformations
(e.g. a human moving), or even swapping an instance of the object with another
(e.g. two different humans).

In order to express these correspondences, we consider an embedding function
$e:S\rightarrow\mathbb{R}^{D}$ associating each 3D point $X\in S$ to the
corresponding $D$-dimensional vector $e_{X}$. Then, for each pixel
$x\in\Omega$ of image $I$, we task a deep network $\Phi$ with computing the
corresponding embedding vector $\Phi_{x}(I)\in\mathbb{R}^{D}.$ From this, we
recover the corresponding canonical 3D point $X\in S$ probabilistically, via a
softmax-like function:

$p(X|x,I,e,\Phi)=\frac{\exp\left(-\langle
e_{X},\Phi_{x}(I)\rangle\right)}{\int_{S}\exp\left(-\langle
e_{X},\Phi_{x}(I)\rangle\right)\,dX}.$ (1)

In this formulation, the embedding function $e_{X}$ is learnable just like the
network $\Phi$. The simplest way of implementing this idea is to approximate
the surface $S$ with a mesh with vertices $X_{1},\dots,X_{K}\in S$, obtaining
a discrete variant of this model:

$p(k|x,I,E,\Phi)=\frac{\exp\left(-\langle
e_{k},\Phi_{x}(I)\rangle\right)}{\sum_{k=1}^{K}\exp\left(-\langle
e_{k},\Phi_{x}(I)\rangle\right)},$ (2)

where $e_{k}=e_{X_{k}}$ is a shorthand notation of the embedding vector
associated to the $k$-th vertex of the mesh and $E$ is the $K\times D$ matrix
with all the embedding vectors $\\{e_{k}\\}_{k=1}^{K}$ as rows.

Given a training set of triplets $(I,x,k)$, where $I$ is an image, $x$ a
pixel, and $k$ the index of the corresponding mesh vertex, we can learn this
model by minimizing the cross-entropy loss:

$\mathcal{L}(E,\Phi)=-\operatornamewithlimits{avg}_{(I,x,k)\in\mathcal{T}}\log
p(k|x,I,E,\Phi).$ (3)

We found it beneficial to modify this loss to account for the geometry of the
problem, minimizing the cross entropy between a ‘Gaussian-like’ distribution
centered on the ground-truth point $k$ and the predicted posterior:

$\mathcal{L_{\sigma}}=-\\!\\!\\!\operatornamewithlimits{avg}_{(I,x,k)\in\mathcal{T}}\sum_{q=1}^{K}g_{S}(q;k)~{}\\!\log
p(q|x,I,E,\Phi),~{}~{}~{}g_{S}(q;k)\propto\exp\left(\\!-\frac{1}{2\sigma^{2}}d_{S}(X_{q},X_{k})\\!\right)$
(4)

where $d_{S}:S\times S\rightarrow\mathbb{R}_{+}$ is the _geodesic distance_
between points on the surface.

Figure 2:  Comparison of IUV (DensePose [17]), and CSE-learned mappings
(vertices predicted in the image are shown in color). CSE training produces
smooth seamless predictions with no shrinking effect (the remaining
discontinuities are due to biases in annotations, see the suppl. mat. for
details).

#### Comparison with DensePose.

DensePose [17], which is the current state-of-the-art approach for recovering
dense correspondences, decomposes the canonical model
$S=\cup_{i=1}^{J}\tilde{S}_{i}$ into $J$ non-overlapping patches each
parametrized via a chart $f_{i}:\tilde{S}_{i}\rightarrow[0,1]^{2}$ mapping the
patch $\tilde{S}_{i}$ to the square $[0,1]^{2}$ of local $uv$ coordinates.
DensePose then learns a network that maps each pixel $x$ in an image $I$ of
the object to the corresponding chart index and $uv$ coordinate as
$\Phi_{x}(I)=(i,u,v)\in\\{1,\dots,M\\}\times[0,1]^{2}$.

An issue with this formulation is that the charting $(\tilde{S}_{i},f_{i})$ of
the canonical model $S$ is arbitrary and needs to be defined manually. The
DensePose network $\Phi$ then needs to output, for each pixel $u$, a
probability value that the pixel belongs to one of the $J$ charts, plus
possible chart coordinates $(u,v)\in[0,1]^{2}$ for each patch. This requires
at minimum $J+2$ values for each pixel, but in practice is implemented as a
set of $3J$ channels as different patches use different coordinate predictors.
Our model eq. 2 is a substantial simplification because it only requires the
initial mesh, whereas the embedding matrix $E$ is learned automatically. This
reduces the amount of manual work required to instantiate DensePose, removes
the seams between the different charts $\tilde{S}_{i}$, that are arbitrary,
and, perhaps most importantly, allows to share a single embedding space, and
thus network, between several classes, as discussed below.

###  3.1 Injecting geometric knowledge via spectral analysis

There are three issues with the formulation we have given so far. First, the
embedding matrix $E$ is large, as it contains $D$ parameters for each of the
$K$ mesh vertices. Second, the representation depends on the discretization of
the mesh, so for example it is not clear what to do if we resample the mesh to
increase its resolution. Third, it is not obvious, given two surfaces $S$ and
$S^{\prime}$ for two different object categories (e.g. humans and chimps), how
their embeddings $e$ and $e^{\prime}$ can be related.

We can elegantly solve these issues by interpreting $e_{X}$ as a smooth
function $S\rightarrow\mathbb{R}^{D}$ defined on a manifold $\mathcal{S}$, of
which the mesh is a discretization. Then, we can use the machinery of
_functional maps_ [39] to manipulate the encodings, with several advantages.

For this, we begin by discretizing the representation; namely, given a real
function $r:S\rightarrow\mathbb{R}$ defined on the surface $S$, we represent
it as a $K$-dimensional vector $\mathbf{r}\in\mathbb{R}^{K}$ of samples
$r_{k}=r(X_{k})$ taken at the vertices. Then, we define the _discrete Laplace-
Beltrami operator_ (LBO) $L=A^{-1}W$ where $A\in\mathbb{R}_{+}^{K\times K}$ is
diagonal and $W\in\mathbb{R}^{K\times K}$ positive semi-definite (see below
for details). The interest in the LBO is that it provides a basis for
analyzing functions defined on the surface. Just like the Fourier basis in
$\mathbb{R}^{n}$ is given by the eigenfunctions of the Laplacian, we can
define a ‘Fourier basis’ on the surface $S$ as the matrix
$U\in\mathbb{R}^{K\times K}$ of generalized eigenvectors $WU=AU\Lambda$, where
$\Lambda$ is a diagonal matrix of eigenvalues. Due to $A$, the matrix $U$ is
orthonormal w.r.t. the metric $A$, in the sense that $U^{\top}AU=I$. Each
column $u_{k}$ of the matrix $U$ is a $K$-dimensional vector defining an
eigenfunction on the surface $S$, corresponding to eigenvalue $\lambda_{k}\geq
0$. The ‘Fourier transform’ of function $\mathbf{r}$ is then the vector of
coefficients $\hat{\mathbf{r}}$ such that $\mathbf{r}=U\hat{\mathbf{r}}$.
Furthermore, if we retain in $U$ only the columns corresponding to the $M$
smallest eigenvalues, so that $U\in\mathbb{R}^{K\times M}$, setting
$\mathbf{r}=U\hat{\mathbf{r}}$ defines a smooth (low-pass) function. Sec. A.1
explains how we construct $W$ and $A$ in detail.

This machinery is of interest to us because we can regard the $i$-th component
$(e_{k})_{i}$ of the embedding vectors as a scalar function defined on the
mesh. Furthermore, we expect this function to _vary smoothly_ , meaning that
we can express the overall code matrix as $E=U\hat{E}$, where
$\hat{E}\in\mathbb{R}^{M\times D}$ is a matrix of coefficients _much_ smaller
than $E\in\mathbb{R}^{K\times D}$. In other words, we have used the geometry
of the mesh to dramatically reduce the number of parameters to learn. LBO
bases can also be used to relate different meshes, as explained next.

Figure 3:  Initializing animal predictor from a human-trained network and mesh
mappings by setting $\hat{E}^{\prime}=C\hat{E}$ for each animal class (see 3.3
for details). For each image-mesh pair, the color coding corresponds to the
new target animal category, as displayed on the corresponding 3D model.

###  3.2 Relating different categories

Let $S$ and $S^{\prime}$ be the canonical shapes of two objects, e.g. human
and chimp. The shapes are analogous, but also sufficiently different to
warrant modelling them with related but distinct canonical shapes. In the
discrete setting, a _functional map_ (FM) [39] is just a linear map
$T\in\mathbb{R}^{K^{\prime}\times K}$ sending functions $\mathbf{r}$ defined
on $S$ to functions $\mathbf{r}^{\prime}=T\mathbf{r}$ defined on $S^{\prime}$.
The space of such maps is very large, but we are interested particularly in
two kinds. The first are point-to-point maps, which are analogous to
permutation matrices $\Pi$, and thus express correspondences between the two
shapes. The second kind are maps restricted to low-pass functions on the two
surfaces. Assuming that functions can be written as
$\mathbf{r}=U\hat{\mathbf{r}}$ and
$\mathbf{r}^{\prime}=U^{\prime}\hat{\mathbf{r}}^{\prime}$ for LBO bases
$(U,A)$ and $(U^{\prime},A^{\prime})$, the functional map
$\mathbf{r}^{\prime}=T\mathbf{r}$ can be written in an equivalent manner as
$\hat{\mathbf{r}}^{\prime}=C\hat{\mathbf{r}}$ where
$C=(U^{\prime})^{\top}A^{\prime}TU$ acts on the spectra of the functions. The
advantage of the spectral representation is that the ${M^{\prime}\times M}$
matrix $C$ is generally much smaller than the $K^{\prime}\times K$ matrix $T$.
Then, we can relate positional embeddings $E$ and $E^{\prime}$ for the two
shapes, which are smooth, as $\hat{E}^{\prime}\approx C\hat{E}$.

When shapes $S$ and $S^{\prime}$ are approximately isometric, we can resort to
automatic methods to establish correspondences $\Pi$ (or $C$) between them.
When $S$ and $S^{\prime}$ are not, this is much harder. Instead, we start from
a small number of manual correspondences $(k_{j},k_{j}^{\prime})$,
$j=1,\dots,Q$ between surfaces and interpolate those using functional maps. To
do this, we use a variant of the ZoomOut method [34] due to its simplicity.
This amounts to alternating two steps: given a matrix $C$ of order
$M_{1}\times M_{1}$, we decode this as a set of point-to-point correspondences
$(k_{j},k_{j}^{\prime})$; then, given $(k_{j},k_{j}^{\prime})$, we estimate a
matrix $C$ of order $M_{2}>M_{1}$, increasing the resolution of the match.
This is done for a sequence $M_{t}=12,16,\dots,256$ until the desired
resolution is achieved (see Sec. A.2 for details).

category | LVIS | training set | test set
---|---|---|---
# inst. | # inst. | # corresp. | coverage | # inst. | # corresp. | coverage
dog | 2317 | 483 | 1424 | 21.5% | 200 | 596 | 10.1%
cat | 2294 | 586 | 1720 | 23.9% | 200 | 591 | 10.0%
bear | 776 | 98 | 289 | 4.8% | 200 | 589 | 9.0%
sheep | 1142 | 257 | 765 | 13.3% | 200 | 511 | 10.2%
cow | 1686 | 426 | 1267 | 19.7% | 200 | 593 | 9.9%
horse | 2299 | 605 | 1783 | 25.8% | 200 | 587 | 10.0%
zebra | 1999 | 665 | 1968 | 28.8% | 200 | 592 | 10.6%
giraffe | 2235 | 651 | 1936 | 27.4% | 200 | 594 | 10.2%
elephant | 2242 | 670 | 1992 | 28.1% | 200 | 592 | 10.0%

Table 1:  Annotation statistics of the DensePose-LVIS dataset. ‘Coverage’ is
expressed as the number of vertices in a given class mesh with at least one
corresponding ground truth annotation. The corresponding animal meshes are
shown on the right (source: hum3d.com).

###  3.3 Cross-species DensePose with functional maps

We are now ready to describe how functional maps can be used to facilitate
transferring and sharing a single DensePose predictor $\Phi$ between
categories with different canonical shapes $S,S^{\prime}$ and corresponding
per-vertex embeddings $E,E^{\prime}$.

To this end, assume that we have learned the pose regressor $\Phi$ on a source
category $(S,E)$ (e.g. humans). We are free to apply $\Phi$ to an image
$I^{\prime}$ of a different category $(S^{\prime},E^{\prime})$ (e.g. chimps),
but, while we might know $S^{\prime}$, we do not know $E^{\prime}$. However,
if we assume that the regressor $\Phi$ can be shared among categories, then it
is natural to also _share their positional embeddings_ as well. Namely, we
assume that $E^{\prime}$ is approximately the same as $E$ up to a remapping of
the embedding vectors from shape $S$ to shape $S^{\prime}$. Based on the last
section, we can thus write $\hat{E}^{\prime}=C\hat{E}$, or, equivalently,
$E^{\prime}=TE$ where $T=U^{\prime}CU^{\top}A$. With this, we can simply
replace $E$ with $E^{\prime}$ in eq. 2 to now regress the pose of the new
category using the same regressor network $\Phi$. For training, we optimise
the same cross entropy loss $\mathcal{L}(E,\Psi)$ in eq. 3, just combining
images and annotations from the two object categories and swapping $E$ and
$E^{\prime}$ depending on the class of the input image.

The procedure above can be easily generalised to any number of categories
$S^{1},\dots,S^{\mathcal{K}}$ with reference to the same source category $S$
and functional maps $C^{1},\dots,C^{\mathcal{K}}$. In our case, we select
humans as source category as they contain the largest number of annotations.

##  4 Datasets

We rely on the DensePose-COCO dataset [17] for evaluation of the proposed
method on the human category and comparison with the DensePose (IUV) training.
For the multi-class setting, we make use of a recent DensePose-Chimps [44]
test benchmark containing a small number of annotated correspondences for
chimpanzees. We split the set of annotated instances of [44] into 500 training
and 430 test samples containing 1354 and 1151 annotated correspondences
respectively.

Additionally, we collect correspondence annotations on a set of 9 animal
categories of the LVIS dataset [21]. Based on images from the COCO dataset
[31], LVIS features significantly more accurate object masks. We refer to this
data as DensePose-LVIS. The annotation statistics for the collected animal
correspondences are given in Table 1. Note that compared to the original
DensePose-COCO labelling effort that produced 5 million annotated points for
the human category ($96\%$ coverage of the SMPL mesh), our annotations are
three orders of magnitude smaller and only $18\%$ of vertices of animal
meshes, on average, have at least one ground truth annotation.

| architecture | AP | $\textbf{AP}_{50}$ | $\textbf{AP}_{75}$ | $\textbf{AP}_{M}$ | $\textbf{AP}_{L}$ | AR | $\textbf{AR}_{50}$ | $\textbf{AR}_{75}$ | $\textbf{AR}_{M}$ | $\textbf{AR}_{L}$
---|---|---|---|---|---|---|---|---|---|---|---
IUV (baselines) | DP-RCNN (R50) [17] | 54.9 | 89.8 | 62.2 | 47.8 | 56.3 | 61.9 | 93.9 | 70.8 | 49.1 | 62.8
DP-RCNN (R101) [17] | 56.1 | 90.4 | 64.4 | 49.2 | 57.4 | 62.8 | 93.7 | 72.4 | 50.1 | 63.6
Parsing-RCNN [56] | 65 | 93 | 78 | 56 | 67 | – | – | – | – | –
AMA-net [20] | 64.1 | 91.4 | 72.9 | 59.3 | 65.3 | 71.6 | 94.7 | 79.8 | 61.3 | 72.3
DP-RCNN* (R50) | 65.3 | 92.5 | 77.1 | 58.6 | 66.6 | 71.1 | 95.3 | 82.0 | 60.1 | 71.9
DP-RCNN* (R101) | 66.4 | 92.9 | 77.9 | 60.6 | 67.5 | 71.9 | 95.5 | 82.6 | 62.1 | 72.6
DP-RCNN-DeepLab* (R50) | 66.8 | 92.8 | 79.7 | 60.7 | 68.0 | 72.1 | 95.8 | 82.9 | 62.2 | 72.4
DP-RCNN-DeepLab* (R101) | 67.7 | 93.5 | 79.7 | 62.6 | 69.1 | 73.6 | 96.5 | 84.7 | 64.2 | 74.2
CSE | DP-RCNN* (R50) | 66.1 | 92.5 | 78.2 | 58.7 | 67.4 | 71.7 | 95.5 | 82.4 | 60.3 | 72.5
DP-RCNN* (R101) | 67.0 | 93.8 | 78.6 | 60.1 | 68.3 | 72.8 | 96.4 | 83.7 | 61.5 | 73.6
DP-RCNN-DeepLab* (R50) | 66.6 | 93.8 | 77.6 | 60.8 | 67.7 | 72.8 | 96.5 | 83.1 | 62.1 | 73.5
DP-RCNN-DeepLab* (R101) | 68.0 | 94.1 | 80.0 | 61.9 | 69.4 | 74.3 | 97.1 | 85.5 | 63.8 | 75.0

Table 2: Performance on DensePose-COCO, with IUV (top) and CSE (bottom)
training (GPSm scores, minival). First block: published SOTA DensePose
methods, second block: our optimized architectures + IUV training, third
block: our optimized architectures + CSE training. All CSE models are trained
with loss $\mathcal{L}_{\sigma}$ (eq. 4), LBO size $M=256$, embedding size
$D=16$.

# LBO basis, $M$ | loss $\mathcal{L}$ | loss $\mathcal{L}_{\sigma}$
---|---|---
32 | 62.9 | 63.2
64 | 64.1 | 63.9
128 | 65.2 | 65.4
256 | 65.6 | 66.1
512 | 65.6 | 65.9
1024 | 65.7 | 65.9

# embedding, $D$ | loss $\mathcal{L}$ | loss $\mathcal{L}_{\sigma}$
---|---|---
2 | 38.3 | 46.4
4 | 60.0 | 64.7
8 | 60.2 | 65.6
16 | 65.6 | 66.1
32 | 65.6 | 66.0
64 | 65.1 | 66.1

training | training mode
---|---
data, % | IUV | loss $\mathcal{L}$ | loss $\mathcal{L}_{\sigma}$
1 | 18.9 | 7.2 | 17.8
5 | 36.6 | 26.5 | 31.4
10 | 42.2 | 36.3 | 39.3
50 | 58.0 | 56.3 | 58.5
100 | 65.3 | 65.4 | 66.1

Table 3:  Hyperparameter search and performance in low data regimes
($\mathbf{AP}$, DensePose-COCO, minival): (left) LBO basis size, $M$ ($D=16$),
(center) embedding size, $D$ ($M=256$), (right) comparison of IUV and CSE
training in small data regimes. DP-RCNN* (R50) predictor.

##  5 Experiments

#### Architectures.

Our networks are implemented in PyTorch within the Detectron2 [54] framework.
The training is performed on 8 GPUs for 130k iterations on DensePose-COCO
(standard s1x schedule [54]) and 5k iterations on DensePose-Chimps and
DensePose-LVIS. The code, trained models and the dataset will be made publicly
available to ensure reproducibility.

Prior to benchmarking the CSE setup, we carefully optimized all core
architectures for dense pose estimation and introduced the following changes
(similarly to [56, 20]): (1) single channel instance mask prediction as a
replacement for the coarse segmentation of [17]; (2) optimized weights for
$(i,u,v)$ components; (3) DeepLab head and Panoptic FPN (similarly to [43]).
The experimental results are reported following the updated protocol based on
GPSm scores [54]. More details on the network architectures and training
hyperparameters are given in the supplementary material.

#### Comparison of CSE vs IUV training.

The comparison of the state-of-the-art methods for dense pose estimation and
our optimized architectures for both IUV and CSE training is provided in Table
2. The CSE-trained models perform better or on par with their IUV-trained
counterparts, while producing a more compact representation ($D=16$ vs $D=75$)
and requiring only simplified supervision (single vertex indices vs $(i,u,v)$
annotations).

#### Influence of hyperparameters.

In Table 3 we investigate the CSE network sensitivity to the size of the LBO
basis, $M$ (left), and the output embedding, $D$ (right), given training
losses $\mathcal{L}$ or $\mathcal{L}_{\sigma}$. The value $M=256$ represents
the tradeoff between mapping’s smoothness and its fidelity. It does not seem
beneficial to increase the embedding size beyond $D=16$, so we adapt this
value for the rest of the experiments. The smoothed loss
$\mathcal{L}_{\sigma}$ yields better performance in a low dimensional setting.

#### Low data regime.

Prior to proceeding with the multi-class experiments on animal classes, we
investigate changes in models performance as a function of the amount of
ground truth annotations by training on subsets of the DensePose-COCO dataset.
As shown in Table 3 on the right, $\mathcal{L}_{\sigma}$-based training scales
down more gracefully and is significantly more robust than $\mathcal{L}$.

#### Multi-surface training.

The results on DensePose-Chimps and DensePose-LVIS datasets are reported in
Tables 4 and 5 (DP-RCNN* (R50), $M=256$, $D=16$). In both cases, training from
scratch results in poor performance, especially in a single class setting.
Initializing the predictor $\Phi$ with the human class trained weights
together with the alignment of mesh-specific vertex embeddings (as described
in 3.3) gives a significant boost. Interestingly, class agnostic training by
mapping all class embeddings to the shared space turns out to be more
effective than having a separate set of output planes for each category (the
latter is denoted as multiclass in Table 5). Quantitative results produced by
the best predictor are shown in Figure 4.

model | $\mathcal{S}_{\textrm{chimp}}$ | $\mathcal{S}_{\textrm{smpl}}$
---|---|---
human model | – | 2.0
loss $\mathcal{L}$ | 8.5 | 3.3
loss $\mathcal{L}_{\sigma}$ | 21.1 | 3.2
cla \+ pretrain $\Phi$ | 36.7 | 34.5
cla \+ align $E$ | 37.2 | 35.7

Table 4: Performance on the DensePose-Chimps dataset with CSE training (AP,
GPSm scores, measured on both chimp and SMPL meshes wrt the GT mapping
$\mathcal{S}_{\textrm{chimp}}\to\mathcal{S}_{\textrm{smpl}}$ from [43]).

model |  cat |  dog |  bear |  sheep |  cow |  horse |  zebra |  giraffe |  elephant | mean
---|---|---|---|---|---|---|---|---|---|---
single class + $\mathcal{L}$ |  5.5 |  4.7 |  1.8 |  0.9 |  2.8 |  4.5 |  12.5 |  11.4 |  19.4 | 7.05
single class + $\mathcal{L}_{\sigma}$ |  20.2 |  16.4 |  10.8 |  14.2 |  22.5 |  24.3 |  23.9 |  27.1 |  26.4 | 20.6
joint training | multiclass + $\mathcal{L}_{\sigma}$ |  20.2 |  18.3 |  19.3 |  25.4 |  22.4 |  26.3 |  33.2 |  30.9 |  29.9 | 25.0
class agnostic + $\mathcal{L}$ |  8.7 |  6.6 |  4.1 |  8.2 |  6.4 |  7.3 |  19.2 |  15.5 |  9.2 | 9.5
class agnostic + $\mathcal{L}_{\sigma}$ |  20.5 |  18.3 |  20.1 |  25.9 |  24.5 |  25.7 |  34.5 |  30.5 |  27.1 | 25.2
cla \+ pretrain $\Phi$ |  28.0 |  28.3 |  22.0 |  31.8 |  36.5 |  32.7 |  43.1 |  41.2 |  34.9 | 33.1
cla \+ align $E$ |  30.9 |  29.4 |  25.1 |  35.3 |  36.5 |  34.3 |  46.0 |  38.3 |  39.6 | 35.0

Table 5: Performance on the DensePose-LVIS dataset with CSE training (AP, GPSm
scores). Figure 4:  Qualitative results on the DensePose-LVIS dataset (single
predictor for all classes).

#### Conclusion.

In this work, we have made an important step towards designing universal
networks for learning dense correspondences within and across different object
categories (animals). We have demonstrated that training joint predictors in
the image space with simultaneous alignment of canonical surfaces in 3D
results in an efficient transfer of knowledge between different classes even
when the amount of ground truth annotations is severely limited.

## Broader impact

In our paper, we help improve the ability of machines to understand the pose
of articulated objects such as humans in images. In particular, we make the
process of learning new object categories much more efficient.

An application of our method is the observation of the human body. This may
come with some concerns on possible negative uses of the technology. However,
we should note that our approach cannot be considered biometrics, because from
pose alone, even if dense, it is not possible to ascertain the identity of an
individual (in particular, we do not perform 3D reconstruction, nor we
reconstruct facial features). This mitigates the potential risk when our
method is applied to humans.

We believe that our work has significant opportunities for a positive impact
by opening up the possibility that machines could ultimately understand the
pose of thousands of animal classes. In addition to numerous applications in
VR, AR, marketing and the like, such a technology can benefit animal-human-
machine interaction (e.g. in aid of the visually impaired), can be used to
better safeguard animals on the Internet (e.g. by detecting animal abuse),
and, perhaps most importantly, can allow conservationists and other
researchers to observe animals in the wild at an unprecedented scale,
automatically analysing their motion and activities, and thus collecting
information on their number, state of health, and other statistics. Thus,
while we acknowledge that this technology may find negative uses (as almost
any technology does), we believe that the positives far outweigh them.

## References

  * [1] Yonathan Aflalo, Anastasia Dubrovina, and Ron Kimmel.  Spectral Generalized Multi-dimensional Scaling.  International Journal of Computer Vision, 118(3):380–392, 2016\.
  * [2] Mykhaylo Andriluka, Umar Iqbal, Anton Milan, Eldar Insafutdinov, Leonid Pishchulin, Juergen Gall, and Bernt Schiele.  PoseTrack: A Benchmark for Human Pose Estimation and Tracking.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5167–5176, 2018.
  * [3] Mykhaylo Andriluka, Leonid Pishchulin, Peter V. Gehler, and Bernt Schiele.  2D Human Pose Estimation: New Benchmark and State of the Art Analysis.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3686–3693, 2014.
  * [4] Mathieu Aubry, Ulrich Schlickewei, and Daniel Cremers.  The wave kernel signature: A quantum mechanical approach to shape analysis.  In IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pages 1626–1633, 2011.
  * [5] Miguel Ángel Bautista, Artsiom Sanakoyeu, Ekaterina Tikhoncheva, and Björn Ommer.  CliqueCNN: Deep Unsupervised Examplar Learning.  In Advances in Neural Information Processing Systems (NIPS), pages 3846–3854, 2016.
  * [6] Benjamin Biggs, Thomas Roddick, Andrew Fitzgibbon, and Roberto Cipolla.  Creatures Great and SMAL: Recovering the Shape and Motion of Animals from Video.  In Asian Conference on Computer Vision (ACCV), pages 3–19, 2018\.
  * [7] Biagio Brattoli, Uta Büchler, Anna-Sophia Wahl, Martin E. Schwab, and Björn Ommer.  LSTM Self-Supervision for Detailed Behavior Analysis.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3747–3756, 2017.
  * [8] Alexander M. Bronstein, Michael M. Bronstein, Leonidas J. Guibas, and Maks Ovsjanikov.  Shape google: Geometric words and expressions for invariant shape retrieval.  ACM Transactions on Graphics (TOG), 30(1):1–20, 2011.
  * [9] Alexander M. Bronstein, Michael M. Bronstein, and Ron Kimmel.  Generalized multidimensional scaling: a framework for isometry-invariant partial surface matching.  Proceedings of the National Academy of Sciences (PNAS), 103(5):1168–1172, 2006.
  * [10] Alexander M. Bronstein, Michael M. Bronstein, Ron Kimmel, Mona Mahmoudi, and Guillermo Sapiro.  A Gromov-Hausdorff framework with Diffusion Geometry for Topologically-Robust Non-rigid Shape Matching.  International Journal of Computer Vision, 89(2–3):266–286, 2010\.
  * [11] Michael M. Bronstein and Iasonas Kokkinos.  Scale-invariant heat kernel signatures for non-rigid shape recognition.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1704 – 1711, 2010.
  * [12] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1302–1310, 2017.
  * [13] Wenzheng Chen, Huan Ling, Jun Gao, Edward J. Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler.  Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer.  In Advances in Neural Information Processing Systems (NeurIPS), pages 9605–9616, 2019.
  * [14] Ronald R. Coifman and Stéphane Lafon.  Diffusion maps.  Applied and Computational Harmonic Analysis, 21(1):5–30, 2006.
  * [15] Asi Elad (Elbaz) and Ron Kimmel.  On Bending Invariant Signatures for Surfaces.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(10):1285–1295, 2003.
  * [16] Danielle Ezuz and Mirela Ben-Chen.  Deblurring and Denoising of Maps between Shapes.  Computer Graphics Forum, 36(5):165–174, 2017.
  * [17] Rıza Alp Güler, Natalia Neverova, and Iasonas Kokkinos.  DensePose: Dense Human Pose Estimation in the Wild.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7297–7306, 2018.
  * [18] Riza Alp Güler, Natalia Neverova, and Iasonas Kokkinos.  DensePose: Dense human pose estimation in the wild.  In Proc. CVPR, 2018.
  * [19] Semih Günel, Helge Rhodin, Daniel Morales, João Campagnolo, Pavan Ramdya, and Pascal Fua.  DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila.  eLife, 2019.
  * [20] Yuyu Guo, Lianli Gao, Jingkuan Song, Peng Wang, Wuyuan Xie, and Heng Tao Shen.  Adaptive Multi-Path Aggregation for Human DensePose Estimation in the Wild.  In ACM International Conference on Multimedia, pages 356–364, 2019\.
  * [21] Agrim Gupta, Piotr Dollár, and Ross Girshick.  LVIS: A Dataset for Large Vocabulary Instance Segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5356–5364, 2019.
  * [22] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi.  Unsupervised Learning of Object Landmarks through Conditional Image Generation.  In Advances in Neural Information Processing Systems (NeurIPS), pages 4020–4031, 2018.
  * [23] Sam Johnson and Mark Everingham.  Clustered Pose and Nonlinear Appearance Models for Human Pose Estimation.  In British Machine Vision Conference (BMVC), pages 1–11, 2010.
  * [24] Sam Johnson and Mark Everingham.  Learning effective human pose estimation from inaccurate annotation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1465–1472, 2011.
  * [25] Angjoo Kanazawa, David W. Jacobs, and Manmohan Chandraker.  WarpNet: Weakly Supervised Matching for Single-View Reconstruction.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3253–3261, 2016.
  * [26] Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik.  Learning category-specific mesh reconstruction from image collections.  In European Conference on Computer Vision (ECCV), pages 386–402, 2018.
  * [27] Artiom Kovnatsky, Michael M. Bronstein, Xavier Bresson, and Pierre Vandergheynst.  Functional correspondence by matrix completion.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 905–914, 2015.
  * [28] Nilesh Kulkarni, Abhinav Gupta, David Fouhey, and Shubham Tulsiani.  Articulation-aware Canonical Surface Mapping.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [29] Nilesh Kulkarni, Shubham Tulsiani, and Abhinav Gupta.  Canonical Surface Mapping via Geometric Cycle Consistency.  In International Conference on Computer Vision (ICCV), pages 2202–2211, 2019.
  * [30] Shuyuan Li, Jianguo Li, Weiyao Lin, and Hanlin Tang.  Amur tiger re-identification in the wild.  arXiv e-prints arXiv:1906.05586, 2019.
  * [31] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: Common Objects in Context.  In European Conference on Computer Vision (ECCV), pages 740–755, 2014.
  * [32] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  SMPL: A skinned multi-person linear model.  ACM Transactions on Graphics (TOG), 34(6):248, 2015.
  * [33] Dominik Lorenz, Leonard Bereska, Timo Milbich, and Björn Ommer.  Unsupervised part-based disentangling of object shape and appearance.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10955–10964, 2019.
  * [34] Simone Melzi, Jing Ren, Emanuele Rodolà, Abhishek Sharma, Peter Wonka, and Maks Ovsjanikov.  ZoomOut: spectral upsampling for efficient shape correspondence.  ACM Transaction on Graphics, 38(6):155:1–155:14, 2019.
  * [35] Tanmay Nath, Alexander Mathis, An Chi Chen, Amir Patel, Matthias Bethge, and Mackenzie Weygandt Mathis.  Using DeepLabCut for 3D markerless pose estimation across species and behaviors.  Nature Protocols, 2019.
  * [36] Natalia Neverova, James Thewlis, Rıza Alp Güler, Iasonas Kokkinos, and Andrea Vedaldi.  Slim DensePose: Thrifty Learning from Sparse Annotations and Motion Cues.  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10915–10923, 2019.
  * [37] Alejandro Newell, Kaiyu Yang, and Jia Deng.  Stacked Hourglass Networks for Human Pose Estimation.  In European Conference on Computer Vision (ECCV), pages 483–499, 2016.
  * [38] David Novotny, Nikhila Ravi, Benjamin Graham, Natalia Neverova, and Andrea Vedaldi.  C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion.  In International Conference on Computer Vision (ICCV), pages 7687–7696, 2019.
  * [39] Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas J. Guibas.  Functional maps: a flexible representation of maps between shapes.  ACM Transactions on Graphics (TOG), 31(4):1–11, 2012.
  * [40] Jonathan Pokrass, Alexander M. Bronstein, Michael M. Bronstein, Pablo Sprechmann, and Guillermo Sapiro.  Sparse modeling of intrinsic correspondences.  Computer Graphics Forum, 32(2):459–468, 2013.
  * [41] Maheen Rashid, Xiuye Gu, and Yong Jae Lee.  Interspecies knowledge transfer for facial keypoint detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6894–6903, 2017.
  * [42] Raif M. Rustamov.  Laplace-Beltrami eigenfunctions for deformation invariant shape representation.  In Symposium on Geometry Processing, pages 225–233, 2007.
  * [43] Artsiom Sanakoyeu, Miguel Ángel Bautista, and Björn Ommer.  Deep unsupervised learning of visual similarities.  Pattern Recognition, 78:331–343, 2018.
  * [44] Artsiom Sanakoyeu, Vasil Khalidov, Maureen S. McCarthy, Andrea Vedaldi, and Natalia Neverova.  Transferring Dense Pose to Proximal Animal Classes.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [45] Saurabh Singh, Derek Hoiem, and David A. Forsyth.  Learning to Localize Little Landmarks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 260–269, 2016.
  * [46] Josef Sivic and Andrew Zisserman.  Video Google: A Text Retrieval Approach to Object Matching in Videos.  In International Conference on Computer Vision (ICCV), pages 1470–1477, 2003.
  * [47] Jian Sun, Maks Ovsjanikov, and Leonidas J. Guibas.  A Concise and Provably Informative Multi-Scale Signature Based on Heat Diffusion.  Computer Graphics Forum, 28(5):1383–1392, 2009.
  * [48] James Thewlis, Samuel Albanie, Hakan Bilen, and Andrea Vedaldi.  Unsupervised learning of landmarks by descriptor vector exchange.  ICCV, 2019.
  * [49] James Thewlis, Hakan Bilen, and Andrea Vedaldi.  Unsupervised Learning of Object Landmarks by Factorized Spatial Embeddings.  In International Conference on Computer Vision (ICCV), pages 3229–3238, 2017.
  * [50] James Thewlis, Hakan Bilen, and Andrea Vedaldi.  Unsupervised object learning from dense invariant image labelling.  In Advances in Neural Information Processing Systems (NIPS), pages 844–855, 2017.
  * [51] Shubham Tulsiani, João Carreira, and Jitendra Malik.  Pose Induction for Novel Object Categories.  In IEEE International Conference on Computer Vision (ICCV), pages 64–72, 2015.
  * [52] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh.  Convolutional Pose Machines.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4724–4732, 2016.
  * [53] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and P. Perona.  Caltech-UCSD Birds 200.  Technical Report CNS-TR-2010-001, California Institute of Technology, 2010\.
  * [54] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [55] Heng Yang, Renqiao Zhang, and Peter Robinson.  Human and sheep facial landmarks localisation by triplet interpolated features.  In IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1–8, 2015.
  * [56] Lu Yang, Qing Song, Zhihui Wang, and Ming Jiang.  Parsing R-CNN for Instance-Level Human Analysis.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 364–373, 2019.
  * [57] Ning Zhang, Jeff Donahue, Ross B. Girshick, and Trevor Darrell.  Part-Based R-CNNs for Fine-Grained Category Detection.  In European Conference on Computer Vision (ECCV), pages 834–849, 2014.
  * [58] Weiyu Zhang, Menglong Zhu, and Konstantinos G. Derpanis.  From Actemes to Action: A Strongly-Supervised Representation for Detailed Action Understanding.  International Conference on Computer Vision (ICCV), pages 2248–2255, 2013.
  * [59] Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, and Honglak Lee.  Unsupervised Discovery of Object Landmarks as Structural Representations.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2694–2703, 2018.
  * [60] Silvia Zuffi, Angjoo Kanazawa, Tanya Y. Berger-Wolf, and Michael J. Black.  Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images "In the Wild".  In International Conference on Computer Vision (ICCV), pages 5358–5367, 2019.
  * [61] Silvia Zuffi, Angjoo Kanazawa, David W. Jacobs, and Michael J. Black.  3D Menagerie: Modeling the 3D Shape and Pose of Animals.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5524–5532, 2017.
  * [62] Silvia Zuffi, Angjoo Kanazawa, David W. Jacobs, and Michael J. Black:.  Lions and Tigers and Bears: Capturing Non-Rigid, 3D, Articulated Shape from Images.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3955–3963, 2018.

##  Appendix A Appendix A

###  A.1 Discrete Laplace-Beltrami operator

#### Laplace-Beltrami operator $(W,A)$.

In order to construct the discrete Laplace-Beltrami Operator (LBO), and with
reference to the notation introduced in the main manuscript, we assume that
the points $X_{k}$ are the vertices of a _simplicial mesh_ , i.e. the union
$\bar{S}=\cup_{f\in F}f$ of a finite set $F$ of triangular faces $f$,
approximating the 3D surface $S$. With slight abuse of notation, we denote
each face $f=(X_{1},X_{2},X_{3})$ as a triplet of vertices oriented in
clockwise order with respect to the normal $N_{f}$ of the face. If we assume
that the function $r$ is continuous and linear within each face, then the
samples $\mathbf{r}$ fully specify the function. Let $b_{1}=X_{3}-X_{2}$,
$b_{2}=X_{1}-X_{3}$ and $b_{3}=X_{2}-X_{1}$, be the edge vectors opposite to
each vertex of the triangle $f$ and let $A_{f}$ be its area. The gradient of
$r$, which is constant on each face $f$, is given by:

$(\nabla r)_{f}=\frac{1}{2A_{f}}\sum_{i=1}^{3}(N_{f}\times
b_{i})r_{f_{i}}=G_{f}\mathbf{r}_{f},~{}~{}~{}\mathbf{r}_{f}=\begin{bmatrix}r_{f_{1}}\\\
r_{f_{2}}\\\ r_{f_{3}}\\\ \end{bmatrix}~{}~{}~{}A_{f}=\frac{1}{2}|b_{1}\times
b_{2}|,~{}~{}~{}N_{f}=\frac{1}{2A_{f}}(b_{1}\times b_{2}).$ (5)

The _Dirichlet energy_ of the function $r$ is the integral of the squared
gradient norm: $\int_{\bar{S}}\|\nabla r\|^{2}\,dS=\sum_{f\in F}A_{f}\|(\nabla
r)_{f}\|^{2}=\sum_{f\in F}\mathbf{r}_{f}^{\top}W_{f}\mathbf{r}_{f},$ where
$A_{f}$ is the area of face $f$ and $W_{f}=A_{f}G_{f}^{\top}G_{f}.$ By summing
$L_{f}$ over the faces, we obtain the overall LBO operator
$W\in\mathbb{R}^{K\times K}$ mapping $\mathbf{r}$ to the total Dirichlet
energy $\mathbf{r}^{\top}W\mathbf{r}$ (since this energy is non-negative, $W$
is positive semi-definite). We also need the diagonal matrix $A$ of _lumped
areas_ , with $A_{kk}$ being a third of the total areas of the triangles
incident on vertex $X_{k}$. 111This is a required normalization factor to
account for the different areas of the triangles w.r.t. the continuous surface
being approximated.

#### Gradient operator $G_{f}$.

We can verify the expression eq. 5 for the gradient as follows. The gradient
dotted with an edge vector $b_{i}$ must give the function change along that
edge. For example, for edge $b_{1}$ we have:

$\langle b_{1},(\nabla r)_{f}\rangle=\sum_{i=1}^{3}\frac{\langle
b_{1},N_{f}\times b_{i}\rangle}{2A_{f}}r_{f_{i}}=\frac{\langle
N_{f},b_{2}\times b_{1}\rangle}{2A_{f}}r_{f_{2}}+\frac{\langle
N_{f},b_{3}\times b_{1}\rangle}{2A_{f}}r_{f_{3}}=r_{f_{3}}-r_{f_{2}}.$

We can write matrix $G_{f}$ in eq. 5 much more compactly as:

$G_{f}=\frac{1}{2A_{f}}\hat{N}_{f}B_{f},~{}~{}~{}~{}~{}B_{f}=B_{f}\hat{\mathbf{1}}^{\top},~{}~{}~{}~{}~{}V_{f}=\begin{bmatrix}X_{1}&X_{2}&X_{3}\end{bmatrix},~{}~{}~{}~{}~{}\hat{a}=\begin{bmatrix}0&-a_{3}&a_{2}\\\
a_{3}&0&-a_{1}\\\ -a_{2}&a_{1}&0\\\ \end{bmatrix},$

Here $\hat{\cdot}$ is the hat operator, such that $a\times b=\hat{a}b$, and
and $\mathbf{1}=(1,1,1)$, so that
$B_{f}=\begin{bmatrix}b_{1}&b_{2}&b_{3}\end{bmatrix}$. By summing $G_{f}$ over
the faces $f$, we obtain an discrete operator $G\in\mathbb{R}^{3|F|\times K}$
mapping the function $\mathbf{r}$ to the gradient in each face.

#### Cotangent weight matrix $W$.

Given the expression for $G_{f}$, we can find a compact expression fo $W_{f}$
in the LBO:

$W_{f}=A_{f}G_{f}^{\top}G_{f}=\frac{1}{4A_{f}}B_{f}^{\top}B_{f}.$

Note that $B_{f}^{\top}\hat{N}_{f}^{\top}\hat{N}_{f}B_{f}=B_{f}^{\top}B_{f}$
because $b_{i}\perp N_{f}$ and thus:

$\langle N_{f}\times b_{i},N_{f}\times b_{j}\rangle=\langle
N_{f},N_{f}\rangle\langle b_{i},b_{j}\rangle-\langle N_{f},b_{j}\rangle\langle
N_{f},b_{i}\rangle=\langle b_{i},b_{j}\rangle.$

$W_{f}$ matches the usual _cotangent discretization_ of the Laplace-Beltrami
operator: $B_{f}$ contains dot products of edges, $2A_{f}$ the norm of their
cross products, and the ratio of these two are cotangents.

#### Divergence operator $D$.

Finally, we sometimes require a divergence operator. For this, let
$X_{f}\in\mathbb{R}^{3}$ a vector defined on each face. In order to compute
the divergence at a vertex $X_{1}$, we find the contour integral of $X_{f}$
along the boundary of the triangle fan centered at $X_{1}$. Thus let
$f=(X_{1},X_{2},X_{3})$ be a face belonging to this fan and
$b_{1},b_{2},b_{3}$ be the corresponding edge vectors as before. The
contribution of this triangle to the contour integral around $X_{1}$ is:

$D_{v}X_{f}=|b_{2}|\cdot\langle n,X_{f}\rangle,~{}~{}~{}n|b_{2}|=b_{2}\times
N_{f}=\frac{1}{2A_{f}}b_{2}\times(b_{3}\times b_{1})=b_{3}\frac{\langle
b_{2},b_{1}\rangle}{2A_{f}}-b_{1}\frac{\langle b_{2},b_{3}\rangle}{2A_{f}}$

###  A.2 Spectra interpolation of correspondences (ZoomOut)

Assume that we have complete correspondences for the mesh $S^{\prime}$, in the
sense that $k^{\prime}_{j}=j,$ $j=1,\dots,K^{\prime}$. We can encode those as
a permutation matrix $\Pi$ such that $\Pi_{ij}=\delta_{k_{i}=j}$, mapping
functions $\mathbf{r}$ on $S$ to function $\mathbf{r}^{\prime}=\Pi\mathbf{r}$
on $S^{\prime}$ (this is analogous to backward warping). This can be rewritten
in ‘Fourier’ space as
$\mathbf{r}^{\prime}=U^{\prime}\hat{\mathbf{r}}^{\prime}=U^{\prime}C\hat{\mathbf{r}}=\Pi\mathbf{r}=\Pi
U\hat{\mathbf{r}},$ which gives us the constraint [16] $U^{\prime}C=\Pi U$. We
can use this equation to find $C$ given $\Pi$, or to find $\Pi$ given $C$.
Finding $\Pi$ is done in a greedy manner, searching, for each row of
$U^{\prime}C$, the best matching row in $U$ (in $L^{2}$ distance). Finding $C$
is done by minimizing $\|U^{\prime}C-\Pi U\|^{2}_{A^{\prime}}$, which results
in $C=(U^{\prime})^{\top}A^{\prime}\Pi U$.

In practice, we found it beneficial to add three more standard constraints
when resolving for $C$. First, let $\Gamma\in\\{0,1\\}^{K\times K}$ be the
symmetry matrix mapping each vertex of mesh $S$ to its symmetric counterpart
(this is trivially determined for our canonical models), and let
$\Gamma^{\prime}$ be the same for $S^{\prime}$. Then a correct correspondence
$\Pi$ between meshes must preserve symmetry, in the sense that
$\Gamma^{\prime}\Pi=\Pi\Gamma$; this constraint can be rewritten in Fourier
space as $\hat{\Gamma}^{\prime}C=C\hat{\Gamma}$, where $\hat{\Gamma}=U\Gamma
U^{\dagger}$. For isometric meshes, the exact same reasoning applies to the
LBO $L=A^{-1}W$ because the LBO is an intrinsic property of the surface (i.e.
invariant to isometry). Our meshes are not isometric, but, after resizing them
to have the same total area, we can use the constraint
$L^{\prime}\Pi\approx\Pi L$ in a soft manner for regularization; it is easy to
show that this reduces to $\Lambda^{\prime}C\approx C\Lambda$ where $\Lambda$
is the matrix of eigenvalues of the LBO. In practices, this encourages $C$ to
be roughly diagonal. Finally, we use the method just described twice, to
estimate jointly a mapping $C$ from mesh $S$ to $S^{\prime}$, and another
$C^{\prime}$ going in the other direction, and enforce $CC^{\prime}\approx I$
(cycle consistency).

##  Appendix B Appendix B

###  B.1 Annotation process

We are following an annotation protocol similar to the one described in the
original DensePose work [18]. We start with instance mask annotations provided
in the LVIS dataset and crop images around each instance. We only annotate
instances with bounding boxes larger than 75 pixels. We do not collect
annotations for body segmentation: instead, the points are sampled from the
whole foreground region represented by the object mask. The annotators are
then shown randomly sampled points displayed on the image and are asked to
click on corresponding points in multiple views rendered from a 3D model
representing the given species. Each worker is asked to annotate 3 points on a
single object instance. The points on the rendered views are mapped directly
to vertex indices of the corresponding model. Each mesh is normalised to have
approximately 5k vertices.

###  B.2 Implementation details

Compared to the original DensePose [17] models, we introduced the following
changes:

  * •

single channel mask supervision as a replacement of the 15-way segmentation;

  * •

RoI pooling size for the DensePose task is set to $28\times 28$;

  * •

a decoder module based on Panoptic FPN, as implemented in [44];

  * •

for the DeepLab models, the head architecture corresponds to [44];

  * •

IUV training, weights of individual loss terms: $w_{mask}=5.0$ (body mask),
$w_{i}=1.0$ (point body indices), $w_{uv}=0.01$ (_uv_ coordinates);

  * •

CSE training, weight on the embedding loss term: $w_{e}=0.6$.

For the DensePose-COCO dataset, all models are trained with the standard s1x
schedule [54] for 130k iterations. On the LVIS and DensePose-Chimps datasets,
the models are trained for 5k iterations with the learning rate drop by the
factor of 10 after 4000k and 4500k iterations.

For the evaluation purposes all 3D meshes are normalised in size to have the
same geodesic distance between the pair of most distant points as the SMPL
model ($P_{\text{dist.,max}}=2.5$). For the animal classes, we do not employ
part specific normalisation coefficients, as done in the updated DensePose
evaluation protocol.

The code, the pretrained models and the annotations for the LVIS dataset will
be publicly released.


* Stassun et al. (2018) Stassun, K. G., Corsaro, E., Pepper, J. A., & Gaudi, B. S. 2018, AJ, 155, 22, doi: 10.3847/1538-3881/aa998a
  * Stassun & Torres (2016) Stassun, K. G., & Torres, G. 2016, AJ, 152, 180, doi: 10.3847/0004-6256/152/6/180
  * Stassun & Torres (2021) —. 2021, ApJ, 907, L33, doi: 10.3847/2041-8213/abdaad
  * Stassun et al. (2019) Stassun, K. G., Oelkers, R. J., Paegert, M., et al. 2019, AJ, 158, 138, doi: 10.3847/1538-3881/ab3467
  * Stock et al. (2023) Stock, S., Kemmer, J., Kossakowski, D., et al. 2023, arXiv e-prints, arXiv:2304.13381, doi: 10.48550/arXiv.2304.13381
  * STScI (2018) STScI. 2018, TESS Input Catalog and Candidate Target List, STScI/MAST, doi: 10.17909/FWDT-2X66
  * Stumpe et al. (2014) Stumpe, M. C., Smith, J. C., Catanzarite, J. H., et al. 2014, PASP, 126, 100, doi: 10.1086/674989
  * Stumpe et al. (2012) Stumpe, M. C., Smith, J. C., Van Cleve, J. E., et al. 2012, PASP, 124, 985, doi: 10.1086/667698
  * Suárez Mascareño et al. (2015) Suárez Mascareño, A., Rebolo, R., González Hernández, J. I., & Esposito, M. 2015, MNRAS, 452, 2745, doi: 10.1093/mnras/stv1441
  * Sullivan et al. (2015) Sullivan, P. W., Winn, J. N., Berta-Thompson, Z. K., et al. 2015, ApJ, 809, 77, doi: 10.1088/0004-637X/809/1/77
  * Terrien et al. (2022) Terrien, R. C., Keen, A., Oda, K., et al. 2022, ApJ, 927, L11, doi: 10.3847/2041-8213/ac4fc8
  * Theano Development Team (2016) Theano Development Team. 2016, arXiv e-prints, abs/1605.02688.  http://arxiv.org/abs/1605.02688
  * Thiabaud et al. (2014) Thiabaud, A., Marboeuf, U., Alibert, Y., et al. 2014, A&A, 562, A27, doi: 10.1051/0004-6361/201322208
  * Tilley et al. (2019) Tilley, M. A., Segura, A., Meadows, V., Hawley, S., & Davenport, J. 2019, Astrobiology, 19, 64, doi: 10.1089/ast.2017.1794
  * Trotta (2008) Trotta, R. 2008, Contemporary Physics, 49, 71, doi: 10.1080/00107510802066753
  * Twicken et al. (2018) Twicken, J. D., Catanzarite, J. H., Clarke, B. D., et al. 2018, PASP, 130, 064502, doi: 10.1088/1538-3873/aab694
  * Van Eylen & Albrecht (2015) Van Eylen, V., & Albrecht, S. 2015, ApJ, 808, 126, doi: 10.1088/0004-637X/808/2/126
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Oliphant, T. E., et al. 2020, Nature Methods, 17, 261, doi: 10.1038/s41592-019-0686-2
  * Wang et al. (2018) Wang, H. S., Lineweaver, C. H., & Ireland, T. R. 2018, Icarus, 299, 460, doi: 10.1016/j.icarus.2017.08.024
  * Weiss et al. (2018) Weiss, L. M., Marcy, G. W., Petigura, E. A., et al. 2018, AJ, 155, 48, doi: 10.3847/1538-3881/aa9ff6
  * Wunderlich et al. (2019) Wunderlich, F., Godolt, M., Grenfell, J. L., et al. 2019, A&A, 624, A49, doi: 10.1051/0004-6361/201834504
  * Zechmeister & Kürster (2009) Zechmeister, M., & Kürster, M. 2009, A&A, 496, 577, doi: 10.1051/0004-6361:200811296
  * Zechmeister et al. (2020) Zechmeister, M., Reiners, A., Amado, P. J., et al. 2020, SERVAL: SpEctrum Radial Velocity AnaLyser, Astrophysics Source Code Library, record ascl:2006.011.  http://ascl.net/2006.011
  * Zeng & Sasselov (2013) Zeng, L., & Sasselov, D. 2013, PASP, 125, 227, doi: 10.1086/669163
  * Zeng et al. (2016) Zeng, L., Sasselov, D. D., & Jacobsen, S. B. 2016, ApJ, 819, 127, doi: 10.3847/0004-637X/819/2/127
  * Zeng et al. (2019) Zeng, L., Jacobsen, S. B., Sasselov, D. D., et al. 2019, Proceedings of the National Academy of Science, 116, 9723, doi: 10.1073/pnas.1812905116
  * Zieba et al. (2023) Zieba, S., Kreidberg, L., Ducrot, E., et al. 2023, Nature, 620, 746, doi: 10.1038/s41586-023-06232-z